New Spring Boot (blue/green) deployment with database access fails to start

[mkheck]

Describe the bug
Initial (blue) deployment of a Spring Boot 3.1.x app accessing an existing MySQL db in Azure using a Service Connector works. Creating a second (green) deployment and Service Connector results in a failed application start, complaining that it can't find the database.

To Reproduce
Steps to reproduce the behavior:

1. Clone the repo at https://github.com/mkheck/azure-spring-apps-training
2. Complete the code in chapter 7 for weather-service and build it with ./mvnw clean package -DskipTests
3. Deploy it per chapter 7
4. Follow these steps to get the required MySQL details for use in creating the Service Connector for the new (green) deployment:

export AZ_RESOURCE_GROUP=<your_resource_group>
export AZ_SPRING_APPS_NAME=<your_asa_instance>
export MYSQL_TARGET_ID="$(az mysql server list --query '[0].id' -g $AZ_RESOURCE_GROUP)"
export MYSQL_SERVERNAME=$(az mysql server list -g $AZ_RESOURCE_GROUP --query '[0].name' -o tsv)
export MYSQL_USERNAME="$(az mysql server list -g $AZ_RESOURCE_GROUP --query '[0].administratorLogin' -o tsv)@${MYSQL_SERVERNAME}"
export MYSQL_HOST="$(az mysql server list -g $AZ_RESOURCE_GROUP --query '[0].fullyQualifiedDomainName' -o tsv)"
env | grep MYSQL

The last command above just verifies the env vars were successfully populated, of course.

Configure the Service Connector for the green deployment:

az spring connection create mysql --app weather-service --deployment green --client-type springBoot --database $MYSQL_SERVERNAME --server $MYSQL_HOST --tg $AZ_RESOURCE_GROUP -g $AZ_RESOURCE_GROUP --service $AZ_SPRING_APPS_NAME

az spring connection create mysql --app weather-service --deployment green --client-type springBoot --database $MYSQL_SERVERNAME --server $MYSQL_HOST --tg $AZ_RESOURCE_GROUP -g $AZ_RESOURCE_GROUP --service $AZ_SPRING_APPS_NAME

az spring app deploy -n weather-service -d green --artifact-path target/demo-0.0.1-SNAPSHOT.jar -s $AZ_SPRING_APPS_NAME -g $AZ_RESOURCE_GROUP

Expected behavior
Preferably, a Service Connector should work for all deployments unless a particular deployment is specified. That would have eliminated the need to define a second Service Connector and should have avoided this issue entirely. That seems to have been the case in earlier versions of ASC/ASA.

Failing that, performing the above steps should result in successful startup of the new (green) deployment.

Screenshots
Please contact if you need any screenshots, but visually there isn't much to see: only a failed startup in the Azure Portal.

Additional context

Can we contact you for additional details? Y/N

Absolutely, always happy to help.

If yes, please send us your contact information to [email protected] and include the issue number in the email title.

[bowen5]

Hi @mkheck , thanks for reporting this issue. This should be a similar issue as mentioned in #40 which contains more detailed discussioned.

Currently it is by designed that Service Connector is landed on Deployment layer, which result in that each deployment is required to config Service Connector and cannot shared. May I know whether this could be a workaround for your current scenario?

At the meantime, we are aware of this experience pain points, and we are working on enhancements to improve and reduce such kind of duplicate configurations. Will update once we have a clear roadmap.

[mkheck]

Hi @bowen5,

This issue is similar to the other one (#40), but not quite the same.

As it stands currently, it seems that a Service Connector can only be created for an existing deployment...which is a "catch 22". So the cycle as it stands now - for a Spring Boot using a database, for example - looks something like this:

1. Create main (default) deployment
2. Create Service Connector (SC)
3. Create second (green) deployment - fails to start, no connector/props/database
4. Create SC for green deployment
5. Try to restart or redeploy - fails

It seems like a SC can only be properly created for a working deployment...and the (green) deployment doesn't work without a Service Connector, at least in my examples and in the configuration/versions I'm testing.

After giving it some more thought, the only thing I can think that might work is not a workable solution for any normal production environment:

1. Remove/disable all code referencing external services in the "ready for production" app to deploy as a secondary deployment (green in this case)
2. Build it (now an island)
3. Deploy it
4. Create the service connector
5. Re-enable/restore all code referencing external services in the app code
6. Build the app (with db dependencies, properties, code, tests, etc.
7. Redeploy

This is nearly (or completely in some cases) unworkable for someone evaluating ASA and will likely discourage new customers from even considering it for production workloads. 😞

I hope this makes sense, but if it doesn't, please let me know. Happy to clarify and help in any way to reduce and/or remove friction for customers and potential adopters!

Thanks so much Bowen.

[bowen5]

Thank you @mkheck for sharing those details!

Yes your observation is exactly correct that the key point of existing painful experience is "SC can only be properly created for a working deployment", that is the area we are working on to improve, to ensure there are SC can be used before doing a real deployment.

Before the improvement shipped, for the workaround would you please try to:

1. Use https://learn.microsoft.com/en-us/cli/azure/spring/app/deployment?view=azure-cli-latest#az-spring-app-deployment-create to create the "green" deployment without any of your real code, then it will use a "hello world" application inside
2. Create service connector
3. Deploy your own app code (no need to disable & re-enable codes using external services)

Would you please check whether above can be a workaround for now? And at the meantime let us improve the experience soon.

[mkheck]

Hi @bowen5, that is exactly what I tried on the advice of Xiangyang Yu, my apologies for the copy/paste error in my initial description. 🫢 The first statement should have been something like this:

az spring app deployment create --name green --app weather-service --runtime-version Java_17 -s $AZ_SPRING_APPS_NAME -g $AZ_RESOURCE_GROUP

Again, sorry for the confusion.