December 23, 2019
Read more ...
Results for
PCF
Overview
Developers, who have been using PCF, have always been asking this question that how do we know which version is running on PCF. We used to check this in on-premise by going to that VM and check the jar version. Also, how to do rollback smoothly for PCF deployments as doing it at SCM (Github) or CI/CD level is painful and risk-prone. It involves verification, approval and other deployment processes.
With PCF 2.6, Pivotal has come up with a feature called "App Revisions". Using this feature, we can rollback the deployment for an application very easily. In this article, we will try to understand this feature and walk through the steps for rollback.
What is App Revisions -
A revision represents code and configuration used by an app at a specific time. In PCF terms, it is an object which will contain references to a droplet, environment variables, and custom start command. Have a look at a simple example below:
In the sample above, we have retrieved all the revisions for an application based on it's GUID using PCF CAPI. In line 15, it has a "resources" object. This object contains all the revisions for an application. Resources will have an array of objects and each object will have a guid which represents a revision. The "description" node will provide a PCF configured description of the revision. For example, in this case, the first revision description at line 30 tells that it is the "Initial version" which got deployed for this application.
Now, whenever there is a change in the application deployment, it will create a new revision with a new GUID. Following are the events which will trigger the new revision:
- A new droplet is created which happens when you do "cf push" after making a change to the code.
- An environment variable is added or changed for the application.
- A custom start command is added or changed in the manifest file.
- An app rolls back to a previous version.
Steps to Follow to Enable, View, and Rollback Revisions -
Step 1 - Take an application that can run on PCF. Deploy it on PCF using cf push command.
Step 2 - Now, get the guid of the running app by executing this command:
Step 3 - Run this command to enable the revision for this app:
Step 4 - Let's check what is the current revision for this deployed application. You will notice an "Initial revision" as a description for the first revision after enabling it:
Step 5 - Now, we will use the environment variable event to trigger the new revision for this example. Add an environment variable to the application using below command and then restage the application to reflect the changes:
Step 6 - Lets, check if there is any new revision that has been added for the application. You will notice a new guid is added in the resources node with a description "New droplet deployed. New environment variables deployed.":
That's the way it keep creating new revisions for each event mentioned above in the article.
Step 7 - If we observe that the description node of the revision is predefined text configured by PCF. We don't have control to change it. Let's suppose we do multiple "New Droplet deployed" type of events. How would we be able to recognize the particular revision? To overcome this, we need to tag each revision with a proper Release version and description. We can use the "metadata" object for this purpose. PCF v3 CAPI provides access to revisions to update the metadata object using the CAPI PATCH request. See the example below:
You will see output like below:
It has been observed that the key_value under metadata does not accept space. It should be Alphanumeric characters only. Though, Empty values are allowed.
Step 8 - Let's set up one more env variable to create one more revision so that we can try to rollback.
Step 9 - Let's now see how to rollback the changes and go back to a previous revision. Going back to the previous version is very easy and can be done with one command:
Step 9 - Let's now see how to rollback the changes and go back to a previous revision. Going back to the previous version is very easy and can be done with one command:
Findings -
- If an environment variable is added in the manifest file and pushed to PCF using "cf push", it will create 2 revisions. One with "New environment variables deployed" and second with "New Droplet Deployed" description. This may create confusion while doing rollback.
To overcome this issue, we can set the environment variable using the "cf set-env" command and then restage the application. This will create only one revision with "New droplet deployed. New environment variables deployed." description. - If you try to roll back to the current version itself, it will not throw any error and create a new revision with a new GUID. This should be raised as a bug to PCF.
- When you do roll back to the previous version, it ensures that it has a currently configured number of instances running. For example, if your application has 2 instances. it will first spin up 1 new instance with revised revision and then destroy the existing instance containers. So, It is ensuring that there is no downtime caused while doing the rollback. However, we need to ensure the capacity for 1 new instance is considered before doing this activity. Also, Users may experience different behavior of the application momentarily as both an old and new version of the apps will be live for a very small period.
- While doing a rollback, keep in mind that by default, PCF retains only the 5 most recent staged droplets in its droplets bucket. So, it cannot go back until this value is increased. Please note every revision does not mean a change in the droplet. PCF retains max 100 revisions by default.
Conclusion
In this article, we have seen how the App Revision feature works and how we can use this feature to do a rollback. I believe, it will further smoothen the process of deployments in PCF. I would recommend to put all these above commands in your CI/CD pipeline and use this feature to tag your deployments in PCF.
July 16, 2019
Read more ...
1. Overview
Most of the developers are creating Microservices these days and deploying to Cloud Platforms. Pivotal Cloud Foundry (PCF) is one of the known cloud platforms. When we talk about deploying applications on PCF, mostly they would be long-running processes which never ends like Web Applications, SPA, REST-based services. PCF monitors all these long-running instances and if one goes down, it spins up the new instance to replace the failed one. This works fine where the process is expected to run continuously but for a batch process, its an overkill. The container will be running all the time with no CPU usage and add up to the cost. Many developers have an ambiguity that PCF cannot run a batch application which can just be initiated based on a request. But that is not correct.
Spring Batch enables to create a batch application and provide many out of the box features to reduce the boilerplate code. Recently, Spring cloud Task has been added in the list of projects to create short-running processes. With both of these options, we can create microservices, deploy on PCF and then stop them so that PCF doesn't try to self heal them. And with the help of PCF Scheduler, we can schedule the task to run them at a certain time of the day. Let's see in this article how we can do that with very few steps.
Spring Batch enables to create a batch application and provide many out of the box features to reduce the boilerplate code. Recently, Spring cloud Task has been added in the list of projects to create short-running processes. With both of these options, we can create microservices, deploy on PCF and then stop them so that PCF doesn't try to self heal them. And with the help of PCF Scheduler, we can schedule the task to run them at a certain time of the day. Let's see in this article how we can do that with very few steps.
2. Pre-requisite
- JDK 1.8
- Spring Boot knowledge
- Gradle
- IDE (Eclipse, VSC, etc...)
- PCF instance
3. Develop the Spring Batch Application
Let's develop a small Spring batch application (spring-batch-master) which will read a file with employee data and then add department name to each of the employee records.
3.1 BatchConfiguration
Let's start with BatchConfiguration file. I have added 2 Jobs here and both show a different way of implementing the batch process. The first one is using Spring Batch Chunks and configured to set up the Job flow with steps. Each step will have reader, processor, and writer configured. The second one is using Tasklet:
public class BatchConfiguration {
public JobLauncher jobLauncher(JobRepository jobRepo) {
SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher();
simpleJobLauncher.setJobRepository(jobRepo);
return simpleJobLauncher;
}
public Job departmentProcessingJob() {
return jobBuilderFactory.get("departmentProcessingJob")
.flow(step1())
.end()
.build();
}
public Step step1() {
return stepBuilderFactory.get("step1")
.<Employee, Employee>chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
public Job job2() {
return this.jobBuilderFactory.get("job2")
.start(this.stepBuilderFactory.get("job2step1")
.tasklet(new Tasklet() {
public RepeatStatus execute
(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
logger.info("Job2 was run");
return RepeatStatus.FINISHED;
}
})
.build())
.build();
}
}
Let's discuss first job departmentProcessingJob in little more detail. This job has step1 which
does read the file, process it and then print it.
3.2 DepartmentReader
This code has logic to read the employee data from a file as part of the first step:
public class DepartmentReader {
public FlatFileItemReader<Employee> reader() {
FlatFileItemReader<Employee> reader = new FlatFileItemReader<Employee>();
reader.setResource(new ClassPathResource("employee_data.txt"));
reader.setLineMapper(new DefaultLineMapper<Employee>() {{
setLineTokenizer(new DelimitedLineTokenizer() {{
setNames(new String[]{"id", "employeenumber", "salary"});
}});
setFieldSetMapper(new BeanWrapperFieldSetMapper<Employee>() {{
setTargetType(Employee.class);
}});
}});
return reader;
}
}
3.3 DepartmentProcessor
This code has logic to add Department Name to each employee record; based on some condition:
public class DepartmentProcessor implements ItemProcessor<Employee, Employee> {
public Employee process(Employee item) throws Exception {
if ("1001".equalsIgnoreCase(item.getEmployeeNumber())) {
item.setDepartment("Sales");
} else if ("1002".equalsIgnoreCase(item.getEmployeeNumber())) {
item.setDepartment("IT");
} else {
item.setDepartment("Staff");
}
System.out.println("Employee Details --> " + item.toString());
return item;
}
}
3.4 DepartmentWriter
This code has logic to print the Employee records with appended Department name:
public class DepartmentWriter implements ItemWriter<Employee> {
public void write(List<? extends Employee> items) throws Exception {
List<String> employeeList = new ArrayList<>();
items.forEach(item -> {
String enrichedTxn = String.join(",", item.getId(),
item.getEmployeeNumber(), item.getSalary(),
item.getDepartment());
employeeList.add(enrichedTxn);
});
employeeList.forEach(System.out::println);
}
}
3.5 BatchCommandLineRunner
Now, lets bootstrap the application logic to run as a Batch process. Spring provides CommandLineRunner to enable it:
public class BatchCommandLineRunner implements CommandLineRunner {
JobLauncher jobLauncher;
Job departmentProcessingJob;4.
public void run(String... args) throws Exception {
JobParameters param = new JobParametersBuilder()
.addString("JobID", String.valueOf(System.currentTimeMillis()))
.toJobParameters(); jobLauncher.run(departmentProcessingJob, param);
jobLauncher.run(job2, param);
}
}
4. Deploy Application on PCF
So far we have created a Spring Batch Job. Now let's deploy to PCF.
You just need to package the code and cf push to PCF with a manifest file.
manifest.yaml:
---
applications:
- name: batch-example
memory: 1G
random-route: true
path: build/libs/spring-batch-master-0.0.1-SNAPSHOT.jar
no-hostname: true
no-route: true
health-check-type: none
We will observe that application did start and executed the logic then exited. The application will be shown as crashed in PCF Apps Manager.
In PCF, all the applications run with process type: web.
It expects the application to be running all the time on some web port. However, for Batch application, it is not the case. So let's see how to handle that:
- Stop the application manually.
- Run this Job either manually with cf run-task command or schedule it using PCF Scheduler
5. Start Batch Application with CF CLI
To run the Spring Batch (Boot) application on PCF, we need to run the following command:
cf run-task <APP Name>
".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher"
Now, you can use this command in a Bamboo/Jenkins pipeline to trigger the application with a cron job.
6. Schedule Batch Job with PCF Scheduler
To schedule the Batch Job, we can use PCF Scheduler. PCF Scheduler enables you to create tasks and schedule them using cron expression. We can go to the application in Apps Manager -> Tasks and click on Enable Scheduling to bind the application with PCF Scheduler. Now you can create Job as shown in below picture. For more details on how to use PCF Scheduler, you can read this blog.
7. Batch Application with Spring Cloud Task
Spring has come up with a new project called Spring Cloud Task (SCT). Its purpose is to create short-lived microservices on Cloud platforms. We just need to add a @EnableTask annotation. This will register TaskRepository and creates the TaskExecution which will pick up the Job defined and execute them one by one.
TaskRepository by default uses in-memory DB however, it can support most of the persistence DBs like Oracle, MySQL, PostgreSQL, etc..
So let's develop one more small application (sct-batch-job) with SCT.
Put @EnableTask in Spring Boot Main Class:
public class EmployeeProcessingBatch {
public static void main(String[] args) {
SpringApplication.run(EmployeeProcessingBatch.class, args);
}
}
Add two Jobs in SCTJobConfiguration file:
public class SCTJobConfiguration {
private static final Log logger = LogFactory.getLog(SCTJobConfiguration.class);
public JobBuilderFactory jobBuilderFactory;
public StepBuilderFactory stepBuilderFactory;
public Job job1() {
return this.jobBuilderFactory.get("job1")
.start(this.stepBuilderFactory.get("job1step1")
.tasklet(new Tasklet() {
public RepeatStatus execute
(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
logger.info("Job1 ran successfully");
return RepeatStatus.FINISHED;
}
})
.build())
.build();
}
public Job job2() {
return this.jobBuilderFactory.get("job2")
.start(this.stepBuilderFactory.get("job2step1")
.tasklet(new Tasklet() {
public RepeatStatus execute
(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
logger.info("Job2 ran successfully");
return RepeatStatus.FINISHED;
}
})
.build())
.build();
}
}
That's it. So if you notice, I have used @EnableBatchProcessing to integrate SCT with Spring Batch. That way we can run Spring Batch application as a Task. We can now push this application to PCF as we did the earlier one and then either run it manually or schedule it using PCF Scheduler.
8. NodeJS Batch Job Scheduling with PCF Scheduler
Similarly, if we have a Batch Job running with NodeJS, we can first stop the application and then create the task with a command to start the nodejs application.
9. Conclusion
In this article, we have talked about how a Spring Batch application can be run on PCF. We can also use Spring Cloud Task to run a short-lived microservices. Spring Cloud Task also provides integration with Spring Batch so you can use full benefits of Batch as well as Spring Cloud Task.
March 27, 2019
Read more ...
Subscribe to:
Posts
(
Atom
)
