Steve, thanks for consolidating the patch. Here're some of my comments and thoughts.
What is essential is that all the existing operations must not change, so that shipping applications do not break.
Yeah, we can retain v1 APIs (this is actually we're doing now), but problem is around "do not break". Does it mean ATS v2 should be compatible with v1 APIs? In other word, do we support that user's old app uses v1 client to talk to v2 server?
is is critical to declare that ATSv1 is stable. Without that guarantee, it is impossible for any application to commit to using the APIs.
Spark depends on this for the SPARK-1537 feature, some ongoing worth with Accumulo depends on this, when Slider adds ATS support we'll depend on this stability guarantee, etc, etc.
I pretty understand the desirability of stable APIs. However, I can see TEZ and Hive/Pig on TEZ started integrating the service even without our declaring the APIs stable. Though the APIs is not declared as stable, it didn't mean we're keeping changing it from release to release. Instead, the reality is that the timeline API is almost compatible since 2.4. Marking it as unstable before is more like reserving the right to change it for improving the service. So I'm not sure if it's good timeline now, as we foresee in the near future, we're going to be upgraded to ATS v2, which may significantly refurnish the APIs.
One area that is not covered in the ATSv1 API is what constitutes a valid entity type or domain?.
Do you mean the mandatory fields? For entity, they're type, id and starttime (which can be optional if the entity containsn at least one event). For event, they are type and timestamp. For domain, they're id.
There is also the fact that the /domain path was added under /ws/v1/timeline/, so matches the path of entity types. Can you have an entity type called "domain"? Was it previously possible?
We cannot. "timeline/domain" blocks the entity type "domain" after domain feature is added. I think we should state it in the documentation (perhaps we wan't to reserve more names for future use). Other than this, I think we shouldn't have any other obligation for naming the identifier.
strictly defining what constitutes a valid entity type via a regular expression, and declaring whether the types are case sensitive.
This is a good idea. We can define the char set and the pattern to prevent users to define random names, but I'm not sure if it is easy to put into practice. The question is whether we're going to break the existing users who have already defined the names that won't match our future regex.
Some comments about the patch:
1. For the bullet points of "Current Status and Future Plans", can we organize them a bit better. For example, we partition them into the groups of a) current status and b) future plans. For bullet 4, not just history, but all timeline data.
2. Can we move "Timeline Server REST API" section before "Generic Data REST APIs"?
3. Application elements table seems to be wrongly formatted. I think that's why site compilation is failed.
4. "Generic Data REST APIs" output examples need to be slightly updated. Some more fields are added or changed.
5. "Timeline Server REST API" output examples are not genuine. Perhaps, we can run a simple MR example job, and get the up-to-date timeline entity and application info to show as the examples.
One additional stuff that is not covered by the documentation is the entity uniqueness. In v1, an entity is globally identified by <type, id>. It means if user1 has posted <type1, id1> in his application, user2 cannot pos the entity with the same identifier in his application even they're completely irrelevant. Therefore, users are suggested to come up with unique entity type for their framework to avoid the namespace collision.