REST API Design Opinions: How Would You Allow Users to Request a Subset of Data?

Communities ColdBox HMVC
1

Hypothetical Question for REST API Design Discussion:

Let’s say you’re building a credit reporting API that can return complete credit report profiles for an individual. The credit profile model is rather large and contains all the details for the customer. For example, personal details, addresses, aliases, risk models (credit score), tradelines (every bank account), etc…

I would probably make the API route look something like this:
(GET) /api/v1/creditProfiles/
The params would be the required information (name, ssn, address, etc), and the response object would contain a mementified CreditProfile object.

Now, what if the API consumer only cares about a subset of the data, like they only want the credit score/risk model instead of the whole massive CreditProfile?

Idea 1: New Nested Endpoint?
Would you create a new REST endpoint like this?
(GET) /api/v1/creditProfiles/riskModel/

Idea 2: Parameters to Control Data Returned
Or would you use the original (GET) /api/v1/creditProfiles/ endpoint but add a special boolean parameter like riskModelOnly=(true/false), which would return just the subset of data instead of the whole model? Similar to how we allow pagination or filter params to be passed in GET requests? I’ve seen a few .NET APIs work this way where you can customize what fields/data you want the endpoint to return.

Idea 3: Cruddy by Design
I’ve watched the “Cruddy by Design” video by Adam Wathan, which makes me think Adam would recommend creating a brand-new handler at the root of the API like this:
(GET) /api/v1/creditRiskModels/
My takeaway from Adam’s video is that he’s not a fan of building folder hierarchies and likes to stick with HTTP verbs as much as possible. In other words, if the action doesn’t match a verb, then create a new handler.

The “Cruddy by Design” approach smells off to me, though. I like the idea of handler names establishing a “domain” where users can easily predict what type of behavior, data, or actions are available by following a logical folder structure. For example, I know that /api/v1/creditProfiles will contain one or more endpoints (or nested endpoints) all related to the “creditProfiles” domain.

What do you think? How would you solve this type of conundrum, and why?

1 Like
2

We approach this from the other direction: REST endpoints should return the smallest possible resource but allow for expandables, in the manner that Stripe uses them in their API:

This dovetails nicely with the Coldbox API handler and mementifier, because all it takes is a little bit of code in your API baseHandler:

// param rc.expand as an array for general-purpose REST field expansion
			event.paramValue( 'expand', [] );
			if ( !isArray( rc.expand ) ) {
				// if it's a string list, i.e. a GET param, convert it to an array
				if ( rc.expand.listLen( ',', false ) > 1 ) {
					rc.expand = rc.expand.listToArray( ',', false );
				}
				else {
					rc.expand = [ rc.expand ];
				}
			}

And then a reference in your endpoint handler when you want to support it:

var thing = wirebox.getInstnace( "someQuickEntity" ).withExpandables( rc.expand )

And then two functions in our custom Quick base entity that allow for invocation of expandables by mapping them to scopes on the object:

/**
	* getExpandedIncludes
	* @return An 'includes' mementifier argument that is the combination of the default
	*
	* @propertyName
	*/
	public array function getExpandedIncludes( array expand = [], array includes ) {
		// not all quick mementos will have an 'expandbale' property since we invented it, so default to an empty array if they don't
		var expandable = this.getDefaultMemento()?.expandable ?: [];

		// remove any expand values we asked for that aren't valid
		var validIncludes = arguments.expand.filter( ( item ) => {
			return expandable.findNoCase( item );
		} );

		// we don't want a default function argument that is calculated at compile time since this.memento isn't iniitialized until instanceReady(), but by the time we actually call this function the instance will be ready
		var initialIncludes = (
			arguments.keyExists( "includes" )
			 ? arguments.includes
			 : this.getDefaultMemento().defaultIncludes
		);

		return (
			len( validIncludes )
			 ? initialIncludes.append( validIncludes, true )
			 : initialIncludes
		);
	}

	/**
	* withExpandables
	* @return A (not-yet-loaded) Quick instance with scopes matching scopeWith{EXPANDABLE} triggered
	*/

	public function withExpandables( array expand = [] ) {
		var expandable = this.getDefaultMemento()?.expandable ?: [];

		// we might want to limit expandables to certain access levels; check for any contexts specific to each access level
		var expandableAuth = this.getDefaultMemento()?.expandableAuth ?: {};

		// remove any expand values we asked for that aren't valid
		var validIncludes = arguments.expand
			.filter( ( item ) => {
				return expandable.findNoCase( item );
			} )
			.filter( ( item ) => {
				return (
					!expandableAuth.keyExists( item ) || _wirebox
						.getInstance( "AuthService" )
						.authorize( context = expandableAuth[ '#item#' ] )
				);
			} );

		// remove any expand values that have auth contexts which we don't pass


		validIncludes.each( ( ex ) => {
			var expandoScope = 'scopeWith' & ex;
			if ( structKeyExists( this, '#expandoScope#' ) ) {
				this[ '#expandoScope#' ]( );
			}
		} );

		return this;
	}

This is a bit different than the question you asked, but as a general matter I think it’s counter-intuitive to have the ‘base resource’ be big and then add parameters to shrink it versus the other way around!

2 Likes
3

This is awesome! What a clever idea!
I like the idea of starting small and allowing the user to “expand” the returned model with properties, relationships, etc.

You could probably also use memento profiles in a similar way where a user would specify the profile they want to get back from the mementifier if you wanted to give the users some presets (e.g. simple, compact, full, etc… )

I’m going to explore this idea in my own API design.

When you document the expand parameter, do you provide a list of available keys the user can provide? I don’t think Swagger files currently support this type of flexibility, but I don’t see why this information couldn’t live in the description field or as a few request/response examples.

4

Yes - we use the @x- extensions in swagger, like this:

	/**
	* /api/v1/family/:familyID
	* @hint View a family record
	* @param-familyID{36} { "schema" : { "type" : "string", "format" : "guid" }, "in" : "path", "description" : "Unique ID of the family to retrieve" }
	* @responses ~/family/get.responses.yml
	* @x-expandable [ "parent1", "parent2", "children", "familyParents" ]
	*/

I can’t remember if we had to change anything in cbswagger or redoc to do that, but if so, it was pretty straightforward. We do the same thing with @x-user-roles and @x-description.

1 Like
5

I know it’s been a little bit, but I have been experimenting with modding my BaseQuickEntity to incorporate some of the expandable functionality you mentioned in your earlier post.

I have a question about how you generate an expanded memento in your handler. Since the getExpandedIncludes() method in your example returns a string, do you have to break the code chain in order to generate your results?

For example:

var thing = wirebox.getInstnace( "someQuickEntity" ).withExpandables( rc.expand );
var includes = thing.getExpandedIncludes( rc.expand );

return thing.asMemento( includes ).get();

I was hoping there would be a way to write a single elegant code chain like this:

var thing = wirebox.getInstnace( "someQuickEntity" )
    .withExpandables( rc.expand ) // implicit memento expansion
    .asMemento()
    .get();

return thing;

Unfortunately, this.memento isn’t initialized until instanceReady(). I’m not sure if there’s a way to work around this limitation.

This is the simplified modification to BaseQuickEntity that I have been testing with in case you or anyone else is curious.

variables._expandable = []; // array of expandable properties

private void function setUpMementifier() {
    var attrs = retrieveAttributeNames( withVirtualAttributes = true );
    
    this.memento = {
        "defaultIncludes" = attrs,
        "mappers" = {},
        "expandable" = variables._expandable
    }

    super.setupMementifier();
}	

public function withExpandables( array expand = [] ) {
    
    var expandable = this.memento.expandable; // each entity has its own expandable list

    // remove any expand values we asked for that aren't permitted for this entity
    var validExpands = arguments.expand.filter( ( item ) => {
        return expandable.findNoCase( item );
    } );

    // call with() to add preload the relationship and append the property to the memento
    validExpands.each( ( ex ) => {
        this.with( ex );
        this.memento.defaultIncludes.append( ex );
    } );

    return this;
}
6

There is a much more straightforward way to handle this. Default the memento to the lowest permissibility and then, if the user has the appropriate permissions, pass in a profile argument to switch the includes. See “Output Profiles” here.

1 Like
7

Dave, you’re correct that it breaks the chainability. Small price to pay.

@jclausen Profiles don’t cut it here because the requirement is for a dynamic, runtime definition of what is desired. We can’t pre-define every permutation of a profile. We do use profiles when we know up front that A, B, and C are different, valid representations, but (much like Stripe’s use case) sometimes we want to say ‘you COULD get any of these five things but that’s up to you.’

2 Likes
8

Thanks @jclausen and @inLeague!

I currently use memento profiles in the majority of my apps, but I wanted to experiment with flexible mementos, as @inLeague specified, similar to Stripe.

I like the idea of returning a minimal memento by default, and allowing the request to expand on the memento as needed.

I spent a little more time experimenting with the objective of not needing to “break the chain” and I found a possible solution:

Here’s my base entity definition:

variables._expandable = [ ]; // allow list of expandable properties

// ...

/**
* withExpandables
* @expand an array of expandable scopes or relationships to load
* @return A (not-yet-loaded) Quick instance with scopes matching with{EXPANDABLE} triggered
*/
public function withExpandables( array expand = [] ) {
	
	// loop through each desired expanded property, and if allowed, add it to the query
	arguments.expand.each( ( item ) => {
		if ( variables._expandable.findNoCase( item ) ) {
			
			// try scopes
			var result = this.tryScopes( item );
			
			// if null, then we try relationships. I don't think there's a way to detect missing relationship.
			if ( isNull( result ) ) {
				result = this.with( item );
			}
			
		};
	} );

	return this;
}

// Post.cfc Entity
variables._expandable = [  "comments" ]; // allow the comments relationship

// Posts.cfc Handler

param rc.expand = ""; // user-specified expandables

prc.page = getInstance( "Post" )
	.withExpandables( listToArray( rc.expand ) )
	.asMemento( includes=rc.expand )
	.get();

So far, the above appears to work. Obviously if you have permissions-based rules for what can/cannot be expanded, you could do the filtering at the handler level, or expand the Base Quick Entity with whatever rules are needed.

If a user passes in a rc.expand value that doesn’t exist in the entity, it will throw an exception saying get#rc.expand# doesn’t exist. You could probably handle this more gracefully… OR (and I am thinking out loud here) if you pass rc.expand as a reference, you could filter it in withExpandables() so that any invalid properties won’t even make it to asMemento().

1 Like