StsAssumeRoleWithWebIdentityCredentialsProvider does not call refreshRequest

[paul-gardiner]

Describe the bug

I am using StsAssumeRoleWithWebIdentityCredentialsProvider to assume an AWS role using a token created on Azure AD/Entra. When the AWS token expires, it will get a new token as long as the WebIdentityToken from Entra has not expired. Once the WebIdentityToken expires, the next refresh to get an AWS token results in an ExpiredTokenException.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When refreshing the AWS token, I expected refreshRequest to be called, which would update the WebIdentityToken. However, it is not called.

Current Behavior

s.a.a.s.s.m.ExpiredTokenException: Token expired: current date/time 1731098245 must be before the expiration date/time 1731095240 (Service: Sts, Status Code: 400, Request ID: 22679931-6a1d-4db3-abd0-a189de181809)
at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:50)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:38)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
at s.a.a.c.i.h.p.s.RetryableStage2.executeRequest(RetryableStage2.java:93)
at s.a.a.c.i.h.p.s.RetryableStage2.execute(RetryableStage2.java:56)
at s.a.a.c.i.h.p.s.RetryableStage2.execute(RetryableStage2.java:36)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingStage.java:53)
at s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingStage.java:35)
at s.a.a.c.i.h.p.s.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at s.a.a.c.i.h.p.s.Api...

Reproduction Steps

Following is the config class (springboot) that creates the StsAssumeRoleWithWebIdentityCredentialsProvider bean.

@Configuration
@Slf4j
class AwsAssumeRoleConfig {

  @Value("${application.aws-role-session-duration}")
  private int sessionDuration;
  @Value("${application.aws-assumed-role-arn}")
  private String assumedRoleArn;

  @Bean
  StsAssumeRoleWithWebIdentityCredentialsProvider assumeRoleWithWebIdentityCredentialsProvider(
      StsClient stsClient, EntraTokenProvider entraTokenProvider) {
    String sessionName = "cct-" + UUID.randomUUID().toString();
    log.info("Creating StsAssumeRole provider for role: {} with session name {} with duration {}",
        assumedRoleArn, sessionName,
        sessionDuration);

    return StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
        .stsClient(stsClient)
        .asyncCredentialUpdateEnabled(true)
        .refreshRequest(assumeRoleRequest -> {
          log.info("entra token requested for assume role");
          OAuth2AccessToken token = entraTokenProvider.getAccessToken();
          if (log.isDebugEnabled()) {
            log.debug("Token received from entra: {}***********",
                token.getTokenValue().substring(0, 5));
          } else {
            log.info("Token received from entra, length {}", token.getTokenValue().length());
          }
          assumeRoleRequest.webIdentityToken(token.getTokenValue());
          assumeRoleRequest.roleArn(assumedRoleArn);
          assumeRoleRequest.roleSessionName(sessionName);
          assumeRoleRequest.durationSeconds(sessionDuration);
        })
        .build();
  }

  @SuppressWarnings("java:S6242")
  @Bean
  StsClient stsClient() {
    return StsClient.builder()
        .region(Region.CA_CENTRAL_1)

        .build();
  }
}

Possible Solution

No response

Additional Information/Context

Dependencies are:

implementation 'org.springframework.cloud:spring-cloud-stream-binder-kinesis:4.0.4'
implementation 'org.apache.httpcomponents.client5:httpclient5'
implementation 'software.amazon.awssdk:kinesis:2.29.1'
implementation 'software.amazon.awssdk:dynamodb:2.29.1'
implementation 'software.amazon.awssdk:cloudwatch:2.29.1'
implementation 'software.amazon.awssdk:sts:2.29.1'
implementation 'org.springframework.security:spring-security-oauth2-client'
implementation 'com.azure.spring:spring-cloud-azure-stream-binder-eventhubs:5.17.1'
implementation 'com.google.protobuf:protobuf-java:4.28.3'

AWS Java SDK version used

2.29.1

JDK version used

OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)

Operating System and version

Oracle Linux Server 8.10

[bhoradc]

Hello @paul-gardiner,

Thank you for reporting the issue.

Firstly, can you please confirm if the following best describes the issue you're experiencing?

1. When the AWS temporary credentials is about to expire, the refreshRequest lambda is invoked to obtain a new web identity token and request new AWS credentials from the AWS STS.
2. However, when the web identity token itself expires, the refreshRequest lambda is not being called, leading to an ExpiredTokenException when attempting to refresh the AWS credentials.

If the above understanding is correct, then this looks an expected behaviour with the StsAssumeRoleWithWebIdentityCredentialsProvider. As I believe the refreshRequest lambda is supposed to be called when the temporary AWS credentials need to be refreshed, regardless of the state of the web identity token.

When the refreshRequest lambda is invoked, isn't the EntraTokenProvider resulting in a fresh web identity token each time, irrespective of the previous token expiration status?

Can you confirm that the EntraTokenProvider implementation is correctly providing a fresh web identity token when called within the refreshRequest lambda, even after the previous token has expired? And if possible, enable the debug logs(redact sensitive info) and share with us?

If my understanding of this problem has any deviation, please provide any additional information or clarification that might help to better understand and troubleshoot this issue.

Regards
Chaitanya

[paul-gardiner]

Not quite. When the AWS credentials expire, if the web identity token is still valid, everything is fine, but if it has expired, it isn't refreshed. My expectation is that StsAssumeRoleWithWebIdentityCredentialsProvider will refresh the web identity token before it expires.

I have worked around the issue by wrapping the StsClient, and then in AssumeRoleWithWebIdentityResponse assumeRoleWithWebIdentity(AssumeRoleWithWebIdentityRequest), I create a new AssumeRoleWithWebIdentityRequest, with a new web identity token. This works, but is a bit messy.

It has been a couple of months, but I'll see if I can get a version going with the issue, and turn on debug.

[phrontizo]

Reopening this thread as I'm seeing the exact same behavior. The problem seems to be that the StsAssumeRoleWithWebIdentityCredentialsProvider.builder().refreshRequest lambda is only ever called once, so a token is obtained from Entra at startup, but that lambda is never invoked again, leading to the use of an expired token.

My solution was to use my own wrapper class and catch the ExpiredTokenException:

    @Override
    public synchronized AwsCredentials resolveCredentials() {
        AwsCredentials result;
        try {
            if (delegate == null) {
                delegate = createDelegate();
            }
            result = delegate.resolveCredentials();
        } catch (ExpiredTokenException e) {
            delegate = createDelegate();
            result = delegate.resolveCredentials();
        }
        return result;
    }

    private StsAssumeRoleWithWebIdentityCredentialsProvider createDelegate() {
        StsClientBuilder stsClientBuilder = StsClient.builder()
                .credentialsProvider(AnonymousCredentialsProvider.create());

        String regionEnv = System.getenv("AWS_REGION");
        if (StringUtils.isNotEmpty(regionEnv)) {
            Region region = Region.of(regionEnv);
            stsClientBuilder = stsClientBuilder.region(region);
        }

        delegate = StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
                .stsClient(stsClientBuilder.build())
                .asyncCredentialUpdateEnabled(true)
                .refreshRequest(assumeRoleRequest -> {
                    System.err.println("[KCL] Refreshing token...");
                    String token = new DefaultAzureCredentialBuilder()
                            .tenantId(tenantId)
                            .build()
                            .getTokenSync(new TokenRequestContext().addScopes(scope)).getToken();
                    System.err.println("[KCL] Token obtained from Azure successfully.");
                    assumeRoleRequest.webIdentityToken(token);
                    assumeRoleRequest.roleArn(roleArn);
                    assumeRoleRequest.roleSessionName(roleSessionName + UUID.randomUUID());
                    assumeRoleRequest.durationSeconds(900);
                })
                .build();
        return delegate;
    }

[sandeepvinayak]

I also faced this issue where the refreshRequest is not doing its intended job. As long as web identity token is valid if sts credentials expire, the refreshRequest gets the new sts credentials. But it doesn't get the new web identity token through a supplied lambda.
here is the code snippet:

StsAssumeRoleWithWebIdentityCredentialsProvider.builder()
                        .stsClient(stsClient)
                        .refreshRequest(r -> {
                            r.roleArn(assumeRole)
                                    .webIdentityToken(overrider.getWebIdentityTokenSupplier().get())  // called on each refresh
                                    .roleSessionName(sessionName);

                            if (overrider.getDurationSeconds() != null) {
                                r.durationSeconds(overrider.getDurationSeconds());
                            }
                        })
                        .build();

overrider.getWebIdentityTokenSupplier().get() gets the new token everytime outside of this workflow.

[YuqiGuo105]

Hi @bhoradc, I investigated this issue and opened #7273. The root cause is that the Consumer overload eagerly builds the request and delegates to the fixed-request overload, so later credential refreshes reuse the same request and web identity token. The PR defers request construction through the existing Supplier and adds a regression test verifying that the lambda is invoked for each refresh (token-1 followed by token-2). The relevant STS tests and Checkstyle pass locally. When you have a chance, I would appreciate your review. Thank you!