Skip to main content

AIP with Sisyphus

To help us follow the Google API Improvement Proposals, Sisyphus implements many standards that are not provided for implementation in the AIP.

AIP-122 Resource Name​

A standard resource name format is provided in the AIP-122 specification and is a unique identifier for representing resources in the system.

Sisyphus generates a special type for all resource name definitions, which is used to easily extract individual parts of the resource name in the code.

message Book {
    option (google.api.resource) = {
        type: "library.googleapis.com/Book"
        pattern: "publishers/{publisher}/books/{book}"
    };

    // Book's resource name
    // formatted as: publishers/{publisher}/books/{book}
    string name = 1;

    // Other fields
}

The above code will additionally generate a Name interface type to represent the resource name of any pattern and generate a concrete implementation type for each pattern that implements the Name interface.

interface Book : Message<Book, MutableBook> {
    // Book's resource name interface type
    public interface Name : ResourceName {
        // The implementation type generated by the specific pattern in the resource name.
        // When the resource name has more than one pattern, multiple implementation types may be generated
        public class Base(`data`: Map<String, String>) : AbstractResourceName(data), Name {
        }
    }

    // The string type of all referenced resource names will also be automatically changed to the corresponding interface type
    val name: Name
}

In addition, if there are fields of type string referenced to the resource name, they will also be automatically changed to the specified interface type, e.g.

message ListBooksRequest {
    // publishers of the book
    // Format: publishers/{publisher_id}
    string parent = 1 [(google.api.resource_reference) = {
        type: "library.googleapis.com/Publisher"
    }];

    // Other fields (e.g. page_size, page_token, filter, etc.)...
}

will change the type of the parent field to Publisher.Name.

interface ListBooksRequest : Message<ListBooksRequest, MutableListBooksRequest> {
    // The string type of the referenced resource name will also be automatically changed to the corresponding interface type
    val parent: Publisher.Name
}

You can create a Name object in Kotlin using Name.of, Name.invoke or Name.tryCreate.

val name = Book.Name.of("publisher_id", "book_id") // just fill in the variable part
val name =
    Book.Name("publishers/123/books/456") // resolve a resource name in its entirety, and automatically create an UnknownResourceName when an unsupported pattern is encountered
val name =
    Book.Name.tryCreate("publishers/123/books/456") // Try to create a resource name, and return null when an unsupported pattern is encountered

When you need to access a specific part of a resource name, you can directly use the corresponding variable name to get the specified part, e.g.

val publisherId = name.publisherId
val bookId = name.bookId

AIP-127 HTTP and gRPC Transcoding​

In the previous section we mentioned how to use transcoding in Sisyphus to access the gRPC API via HTTP/Json.

In most cases, the semantics of the HTTP RequestHeader and the gRPC Metadata are not identical, so when accessing the gRPC API using HTTP, we also need to configure the transformation of the HTTP Header into the gRPC Metadata when accessing the gRPC API using HTTP.

Use the TranscodingHeaderExporter to customize this process.

@Component
class MyTranscodingHeaderExporter : TranscodingHeaderExporter {
    private val keys = mutableMapOf<String, Metadata.Key<String>>()

    private val allowedHeader = setOf(
        HttpHeaders.AUTHORIZATION.lowercase(),
        HttpHeaders.COOKIE.lowercase(),
        HttpHeaders.FROM.lowercase(),
        HttpHeaders.HOST.lowercase(),
        HttpHeaders.REFERER.lowercase(),
        HttpHeaders.ACCEPT_LANGUAGE.lowercase(),
        HttpHeaders.PRAGMA.lowercase(),
    )

    private fun key(header: String): Metadata.Key<String> {
        return keys.getOrPut(header) {
            Metadata.Key.of(header, Metadata.ASCII_STRING_MARSHALLER)
        }
    }

    override fun export(request: ServerRequest, metadata: Metadata) {
        for ((key, values) in request.headers().asHttpHeaders()) {
            val normalized = key.lowercase()
            if (normalized in allowedHeader) {
                metadata.put(key(normalized), values.joinToString(","))
            }
        }
    }
}

In addition, due to the inherent limitations of the gRPC client, gRPC requests cannot be customized with User-Agent, we can also use TranscodingHeaderExporter to convert the HTTP User-Agent to another gRPC Metadata to the gRPC service.

override fun export(request: ServerRequest, metadata: Metadata) {
    for ((key, values) in request.headers().asHttpHeaders()) {
        if (key.equals(HttpHeaders.USER_AGENT, ignoreCase = true)) {
            // Convert User-Agent to X-User-Agent
            metadata.put(key("x-user-agent"), values.joinToString(","))
            continue
        }
        val normalized = key.lowercase()
        if (normalized in allowedHeader) {
            metadata.put(key(normalized), values.joinToString(","))
        }
    }
}

The HTTP/Json interface is typically used in browsers, for which we also need to support preflight requests. Sisyphus will automatically support preflight requests based on the configuration of the supported gRPC methods.

However, when we want to return the custom Metadata of the gRPC response to the client, due to browser limitations, we need to enumerate the possible response headers in the preflight request in order to be read by the front-end.

The TranscodingCorsConfigurationInterceptor interface can be used to customize all preflight responses.

@Component
class MyTranscodingCorsConfigurationInterceptor : TranscodingCorsConfigurationInterceptor {
    override fun intercept(
        config: CorsConfiguration,
        method: ServerMethodDefinition<*, *>,
        pattern: HttpRule.Pattern<*>,
        path: String
    ): CorsConfiguration {
        config.addExposedHeader("request-id")
        return config
    }
}

AIP-132 Standard methods: List​

List interfaces are common in API design, and Sisyphus provides a number of tools to make it easy for developers to quickly create list API.

Sorting​

In the List interface we may need to support the order_by field to specify the sorting rules for the list, the sisyphus-dsl component provides OrderDsl to parse the Ordering syntax in the AIP-132.

val start = OrderDsl.parse("foo desc, bar")

In addition, if JDBC is used as the backend database, the sisyphus-jdbc middleware can translate the ordering syntax into order by part of SQL.

val orderByBuilder = orderByBuilder {
    field("foo", Tables.BAR.FOO)
}

jooq.selectFrom(Tables.BAR).orderBy(orderByBuilder.build(orderBy))

Use the orderByBuilder DSL to quickly embed ordering syntax into a JOOQ query.

Filtering​

Sisyphus also implements a corresponding tool for filters DSL, but since the implementation of filters is more complex, we will cover the details in AIP-160 below.

Pagination​

Sisyphus recommends customizing a Protobuf Message to store the paging context and encode its binary serialization as UrlSafeBase64 as a page_token.

Here are two very common paging contexts for reference.

message IndexAnchorPaging {
    int64 index = 1;
}

message OffsetPaging {
    int32 offset = 1;
}

IndexAnchorPaging is anchor-based paging, can be used for high performance paging, but can not support page skipping, usually used on the client side, its meaning is the index of the last element of the previous page, when getting the next page just need to provide WHERE id > paging.token to get the next page.

OffsetPaging is an offset-based paging, which is worse in large amount of data, but more flexible and can support page skipping, usually used in the management backend, and the SQL statement translated as LIMIT offset, limit is needed to get the next page.

AIP-160 Filtering​

Filtering is an important part of the List interface and Sisyphus has gone to great lengths to simplify the work of developers in this section, providing a very simple DSL to help developers translate the Filtering syntax structure into traditional SQL.

Filter in SQL​

This is the most efficient way to implement filters, leaving the filter to the database to be processed, ensuring that a specified amount of data is returned, as shown in [sisyphus-jdbc](https://github.com/ButterCam/sisyphus/tree/master/middleware/sisyphus- jdbc) The sqlBuilder DSL is used in the middleware to create the TableSqlBuilder.

val sqlBuilder = sqlBuilder(Tables.FOO) {
    field("name", Tables.FOO.ID) {
        Foo.Name.tryCreate(it as String)?.note?.toLong()
    }
    field("bar", Tables.FOO.BAR)
    field("title", Tables.BAZ.TITLE)
    library(object : FilterStandardLibrary() {
        fun withBaz(): Join {
            return Join {
                it.leftJoin(Tables.BAZ).on(Tables.BAZ.FOO_ID.eq(Tables.FOO.ID))
            }
        }
    })
}

Specify the field names that can be used in the filtering statement in the sqlBuilder DSL and map them to fields in the database, as well as specify a custom transformation function.

Use this sqlBuilder to build SQL with JOOQ.

sqlBuilder.select(jooq, filter)
    .orderBy(orderByBuilder.build(orderBy))

And the client just needs to fill in the filter field of the List interface with a filtering statement like name='foos/1' OR baz=2, which automatically translates to

SELECT *
FROM foo
WHERE (id = 1 OR baz = 2)

You can also extend FilterStandardLibrary to provide functions that can be used in filtering statements, and to enable join queries if the function returns a Join object.

When the client fills in the filter field with title='FooBar' withBaz(), Sisyphus applies the Join value of WithBaz to the JOOQ query.

SELECT *
FROM foo
         LEFT JOIN baz ON baz.foo_id = foo.id
WHERE (title = 'FooBar')

When a function returns Condition, the function can be written in a condition, allowing for complex queries that are not possible with filter statements, or limited queries that are wrapped for security.

library(object : FilterStandardLibrary() {
    fun safeQuery(): Condition {
        return Tables.FOO.PASSWORD.eq("123456")
    }
})

When the client fills in the filter field with bar='foobar'' AND safeQuery(), the filter will be converted to the following SQL statement.

SELECT *
FROM foo
WHERE (baz = 2 AND password = '123456')

Filter in Code​

Another type of filtering is code filtering, which means that the data found in the database is logically filtered by local code. The filters in this way will be more powerful, but there may also case empty pages.

Sisyphus uses CEL to describe this filter.

The full name of CEL is Common Expression Language, a non-Turing-complete expression language proposed by Google that can be deeply integrated with Protobuf.

CelEngine is used in sisyphus-dsl to execute CEL expressions.

val celEngine = CelEngine()
val filteredItems = foos.filter {
    celEngine.eval(
        "foo.bar == 'foobar' && foo.title == 'FooBar'",
        mapOf("foo" to it)
    )
}

In addition to being used as a filter, CEL can be used in a variety of scenarios that require dynamic code execution, such as permission checking, dynamic content, etc.

AIP-161 Field masks​

When implementing the Update interface, we often require partial update functionality, where the client can choose to update some fields instead of the entire object. Usually, this can be achieved by using a null value that is not updated, but this makes it impossible to distinguish whether the user wants to delete the field or does not want to update the field.

In this case, FieldMask can be used to solve this problem by passing an additional update_mask field to guide the update operation of the service field.

Foo.resolveMask(updateMask).paths.forEach { field ->
    when (field) {
        Foo.BAR_FIELD_NAME -> this.bar = foo.bar.takeIf { foo.hasBar() }
        Foo.TITLE_FIELD_NAME -> this.title = foo.title.takeIf { foo.hasTitle() }
    }
}

Use the resolveMask method to normalize update_mask and access each field to be updated via paths.