atelet: serialize node-local operations per actor

[Troy Chiu (troychiu)]

Towards #372

Serializes node-local operations (Run, Checkpoint, UploadPausedCheckpoint, Restore, Terminate) on a per-actor basis in atelet to prevent concurrent operations from interfering with each other's on-disk state (e.g. reading or wiping checkpoint directories concurrently).

• Tests pass
• Appropriate changes to documentation are included in the PR

---

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Zoe Zhao (zoez7)]

Do we need this given that the ate apiserver already holds a global lock per actor before each Actor operation?

[Troy Chiu (troychiu)]

I think it's still needed.

If an ateapi request cancels or times out, ateapi exits and releases the distributed lock. However, the in-flight gRPC call on atelet may still be running or unwinding. If a retry arrives immediately, ateapi re-acquires the lock and issues a second RPC to atelet, resulting in concurrent operations interfering with the actor's local state.