RESTful server checks consistency of batch size which breaks the model with arbitrary input or output shape #1047

tobegit3hub · 2018-08-10T10:40:31Z

Bug Report

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow Serving installed from (source or binary): binary
TensorFlow Serving version: 1.8.0

Describe the problem

The RESTful server will check the consistency of batch size which may requires all the input and output tensor to support "batch size". In fact, our models may output the tensor which does not have batch size. These models work like a charm with TensorFlow Serving gRPC APIs but not TensorFlow Serving RESTful APIs.

Exact Steps to Reproduce

This is easy to re-produce and we can export the example SavedModel with these script. The code to break RESTful server is constant_op = tf.constant([1.0, 2.0]) which is a tensor with shape [2].

import os
import tensorflow as tf
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import (
    signature_constants, signature_def_utils, tag_constants, utils)
from tensorflow.python.util import compat

model_path = "model"
model_version = 1

keys_placeholder = tf.placeholder(tf.int32, shape=[None, 1], name="keys")
keys_identity = tf.identity(keys_placeholder, name="inference_keys")

constant_op = tf.constant([1.0, 2.0])

sess = tf.Session()
sess.run(tf.global_variables_initializer())

model_signature = signature_def_utils.build_signature_def(
    inputs={
        "keys": utils.build_tensor_info(keys_placeholder),
    },
    outputs={
        "keys": utils.build_tensor_info(keys_identity),
        "constant": utils.build_tensor_info(constant_op)
    },
    method_name=signature_constants.PREDICT_METHOD_NAME)

export_path = os.path.join(
    compat.as_bytes(model_path), compat.as_bytes(str(model_version)))

builder = saved_model_builder.SavedModelBuilder(export_path)
builder.add_meta_graph_and_variables(
    sess, [tag_constants.SERVING],
    clear_devices=True,
    signature_def_map={
        signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: model_signature,
    })

builder.save()

Then load the model with TensorFlow Serving while exposing both gRPC and RESTful APIs. Remember to change the absolute path of your model.

tensorflow_model_server --port=8501 --rest_api_port=8502 --model_base_path="/foo/bar/model/"

Then request the RESTful APIs with simple curl command.

curl -H "Content-Type: application/json" -X POST -d '{"instances": [{"keys": [[1]]}]}' http://127.0.0.1:8502/v1/models/default:predict

This will get the error message of { "error": "Tensor name: keys has inconsistent batch size: 1 expecting: 2" }. If we change to code of model signature with constant_op = tf.constant([1.0]), this will work because our request just have the batch size of [1].

And we have implement the gRPC client to make sure the model is normal. Here is the code of the gRPC client for this mode.

import time
import numpy
import tensorflow as tf
from grpc.beta import implementations
from tensorflow_serving.apis import predict_pb2, prediction_service_pb2

def main():
  host = "0.0.0.0"
  port = 8501
  model_name = "default"
  model_version = -1
  signature_name = ""
  request_timeout = 10.0

  # Generate inference data
  keys = numpy.asarray([[1]])
  keys_tensor_proto = tf.contrib.util.make_tensor_proto(keys, dtype=tf.int32)

  # Create gRPC client
  channel = implementations.insecure_channel(host, port)
  stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
  request = predict_pb2.PredictRequest()
  request.model_spec.name = model_name
  if model_version > 0:
    request.model_spec.version.value = model_version
  if signature_name != "":
    request.model_spec.signature_name = signature_name
  request.inputs["keys"].CopyFrom(keys_tensor_proto)

  # Send request
  result = stub.Predict(request, request_timeout)
  print(result)

if __name__ == "__main__":
  main()

And this gRPC client will work and output the expected result.

outputs {
  key: "constant"
  value {
    dtype: DT_FLOAT
    tensor_shape {
      dim {
        size: 2
      }
    }
    float_val: 1.0
    float_val: 2.0
  }
}
outputs {
  key: "keys"
  value {
    dtype: DT_INT32
    tensor_shape {
      dim {
        size: 1
      }
      dim {
        size: 1
      }
    }
    int_val: 1
  }
}
model_spec {
  name: "default"
  version {
    value: 1
  }
  signature_name: "serving_default"
}

Source code / logs

Above.

The text was updated successfully, but these errors were encountered:

tobegit3hub · 2018-08-10T10:56:27Z

The root cause is in https://github.com/tensorflow/serving/blob/master/tensorflow_serving/util/json_tensor.cc#L819 . It requires all the named tensors to have the same first dimension.

It is quite unreasonable and inconvenient. In one of our scenarios, we generated the vocabulary(hash table) to map label index and string names and export it as part of the model. This vocabulary obviously does not have the same first dimension as the request data.

This is not the consistent interfaces of the original TensorFlow Serving gRPC APIs and hope this can change.

netfs · 2018-08-10T23:01:44Z

Thanks for the detailed report. The intent of the code was to keep the input
and output symmetric. IOW input tensors for each feature, all need to have
the same batch size. Clearly this does not apply for output.

If you can prepare a PR with the fix i'd be happy to review!

Thanks again.

tobegit3hub · 2018-08-13T03:09:45Z

Thanks for your response @netfs . I can submit the PR to check input tensor only.

But is it possible for users to specify different input shapes? For example, some developers may want to restrict the number of top-N prediction and add [tf.int32] placeholder in SavedModel which has different shape of other input tensors. I have test with the following code to construct model with different-shape inputs and it is supported in TensorFlow Serving.

keys_placeholder = tf.placeholder(tf.int32, shape=[None, 1], name="keys")
another_keys_placeholder = tf.placeholder(tf.int32, shape=[3], name="another_keys")
keys_identity = tf.identity(keys_placeholder, name="inference_keys")
constant_op = tf.constant([1.0, 2.0])

sess = tf.Session()
sess.run(tf.global_variables_initializer())

model_signature = signature_def_utils.build_signature_def(
    inputs={
        "keys": utils.build_tensor_info(keys_placeholder),
        "another_keys": utils.build_tensor_info(another_keys_placeholder),
    },
    outputs={
        "keys": utils.build_tensor_info(keys_identity),
        "constant": utils.build_tensor_info(constant_op)
    },
    method_name=signature_constants.PREDICT_METHOD_NAME)

netfs · 2018-08-14T18:03:01Z

i see what you mean, and its seems plausible that models requiring inputs
each with a different batch size. though this presents a problem expressing
such values in JSON requests.

the JSON request is list of rows of individual inputs, that we internally columnize
(and where the present checks are tripping your usage). we expect each row
of the input to have all named inputs. so a request typically looks as follows
(with a batch_size of 2, for all inputs):

{
  "instances": [ 
    {
       "tag": ["foo"]
       "signal": [1, 2, 3, 4, 5]
       "sensor": [[1, 2], [3, 4]]
    },
    {
       "tag": ["bar"]
       "signal": [3, 4, 1, 2, 5]]
       "sensor": [[4, 5], [6, 8]]
    },
 ] 
}

assuming we had another input say "location_id", that has batch_size=1, how
would such a request look? one way would be:

{
  "instances": [ 
    {
       "tag": ["foo"]
       "signal": [1, 2, 3, 4, 5]
       "location_id": "baz"
    },
    {
       "tag": ["bar"]
       "signal": [3, 4, 1, 2, 5]]
        // missing: "location_id"
    },
 ] 
}

(note missing "location_id" in the second element of instances list)

this can get confusing (or maybe not :-) seeing missing named input
in some rows. we could choose to ignore such missing values, and
let the model processing complain if the batch size is not correct
(it would). i think this should be ok. let me think about this and get
back.

tobegit3hub · 2018-08-15T07:34:01Z

Thanks @netfs for the detailed explaination.

Actually we know about this limitation after reading the format of JSON requests. Since we separate batch data into items in instances array, it requires each item of instances has the same shape. Otherwise, passing optional parameters in different items is really confusing.

Because both the inputs and outputs of TensorFlow models are Tensor, have you considerated using the following format? It is much more compact which may improves performance and more similar with TensorFlow Serving gRPC APIs. We have another RESTful serving called simple_tensorflow_serving which uses this format and be compatible with all TensorFlow SavedModels.

{
  "instances": {
    "tag": ["foo", "bar"],
    "signal": [[1, 2, 3, 4, 5], [3, 4, 1, 2, 5]],
    "sensor": [[[1, 2], [3, 4]], [[4, 5], [6, 8]]]
  }
}

netfs · 2018-08-15T20:01:45Z

yes columnar format is what i am presently leaning towards (maybe under a different key called "inputs").

additionally we need to make sure if batching is enabled:
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_advanced.md#batching

it degrades gracefully when we have inputs with differing 0-th dimension (i.e. batch sizes).

netfs · 2018-08-15T23:18:42Z

My current proposal is as follows:

Continue to support the current row-format of instances and predictions (as the input/output) with the restriction that 0-th dimension of all inputs/outputs should be same (read: consistent batch size). This format is easier to read for common use-cases.
In addition to above row-format, we allow a new column-format style (similar to your proposal) with inputs key in request and outputs key in response. These keys would hold values in column-format. This would be very similar to the gRPC predict API. This would make your request look as follows:

{
  "inputs": {
    "tag": ["foo", "bar"],
    "signal": [[1, 2, 3, 4, 5], [3, 4, 1, 2, 5]],
    "sensor": [[[1, 2], [3, 4]], [[4, 5], [6, 8]]]
  }
}

and response will be:

{
  "outputs": {
    "constant": ...,
    "keys": ...,
  }
}

We do not allow mixing these two formats in the same request. And response format will match with that of input. So you get predictions for instances (row-format) and outputs for inputs (column-format).

This will allow users to choose suitable format to express their input/output for their models.

tobegit3hub · 2018-08-20T08:24:05Z

Great and looking forward to the new column-format 👍

Thanks @netfs . This issue may be closed if we have supported the column-format for arbitrary input or output shapes.

netfs · 2018-08-20T21:52:35Z

thanks for taking a look. i will compose a change to add support for column format and update this thread once the change is ready. will be great if you can help test it, once its ready.

tobegit3hub · 2018-08-24T08:31:12Z

Great! The implementation is nice and we are glad to test it.

There is one problem I found in that commit a5b7cfd . The request json data for columnar inputs should be something like {'inputs': {'constant': [2.0, 3.0, 4.0]}} instead of {'inputs': [2.0, 3.0, 4.0]}, right? @netfs

def testPredictColumnarREST(self):
    """Test Predict implementation over REST API with columnar inputs."""
    model_path = self._GetSavedModelBundlePath()
    host, port = TensorflowModelServerTest.RunServer('default',
                                                     model_path)[2].split(':')
     # Prepare request
    url = 'http://{}:{}/v1/models/default:predict'.format(host, port)
    json_req = {'inputs': [2.0, 3.0, 4.0]}
     # Send request
    resp_data = None
    try:
      resp_data = CallREST('Predict', url, json_req)
    except Exception as e:  # pylint: disable=broad-except
      self.fail('Request failed with error: {}'.format(e))
     # Verify response
    self.assertEquals(json.loads(resp_data), {'outputs': [3.0, 3.5, 4.0]})

netfs · 2018-08-24T18:57:10Z

if 'constant' is the only named input, then you can skip specifying that,
if there are more inputs then you need to use the object notation and
specify each.

see the unit tests a5b7cfd#diff-a8a252f963e32d57426b166ad91b1aea for details.

tobegit3hub · 2018-08-27T01:45:26Z

Thanks @netfs !

request in columnar format. This allows expressing named input (or output) tensors that do not have the same 0-th dimension. This change implements the proposal outline here: tensorflow#1047 (comment) Fixes tensorflow#1047 PiperOrigin-RevId: 209666367

* Fix the reference to the old file name. PiperOrigin-RevId: 209212508 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209212799 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209221439 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209229489 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209234879 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209238558 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209244752 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209252876 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209256393 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209263314 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209265561 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209285595 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209289986 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209302666 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209315060 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209319809 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209393642 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209410842 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209426726 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209435699 * Add unzip to devel package PiperOrigin-RevId: 209441638 * Correct half_plus_two -> half_plus_three in docker.md. PiperOrigin-RevId: 209441832 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209448213 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209459701 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209469391 * Default development Dockerfiles to produce a more optimized ModelServer PiperOrigin-RevId: 209472456 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209479375 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209490489 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209502117 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209510969 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209518990 * Update optimization flags recommendations PiperOrigin-RevId: 209522738 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209525370 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209529568 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209534560 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209538855 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209543217 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209553461 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209559952 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209567477 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209572442 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209595539 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209603479 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209612811 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209624218 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209635989 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209645776 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209655351 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209666113 * Add support to express input and output tensors of Predict request in columnar format. This allows expressing named input (or output) tensors that do not have the same 0-th dimension. This change implements the proposal outline here: tensorflow#1047 (comment) Fixes tensorflow#1047 PiperOrigin-RevId: 209666367 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209676286 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209685647 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209693666 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209700023 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209703988 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209708752 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209712998 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209731825 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209738910 * Enable model version labels in the model server. PiperOrigin-RevId: 209751560 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209753234 * Add build status badges for continuous+nightly builds. PiperOrigin-RevId: 209792937 * Remove status of "TF Head" build. Existing "Ubuntu Build" provides enough signal on the state of the gitrepo. PiperOrigin-RevId: 209799778 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209805263 * Replaced calls to tensorflow::StringPiece::ToString with string conversions. That is, instances of sp.ToString() are replaced with string(sp). This will allow tensorflow::StringPiece::ToString to be removed, which is necessary before it can be replaced with absl::string_view. PiperOrigin-RevId: 209806694 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209814898 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209825089 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209836354 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209845959 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209856090 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209863960 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209870287 * Rename class from HttpRestPredictionHandler -> HttpRestApiHandler, so it be used to for additional REST API methods and not be limited to just prediction APIs. This class is mostly a router from HTTP endpoints to internal worker classes, and is expected to stay small even as other REST API endpoints get added. PiperOrigin-RevId: 209874946 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209875736 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209880622 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209884317 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209894876 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209900704 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209913796 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209936141 * Add missing dependency to tensorflow-serving-api build_pip_package * Meet Alpine Linux requirements for mktemp mktemp TEMPLATE must end with XXXXXX in Alpine Linux % docker run --rm -it alpine:3.6 /bin/sh / # mktemp -d -t libevent.XXXXX mktemp: Invalid argument / # mktemp -d -t libevent.XXXXXX /tmp/libevent.jKecLG / # mktemp --help BusyBox v1.26.2 (2017-11-23 08:40:54 GMT) multi-call binary. Usage: mktemp [-dt] [-p DIR] [TEMPLATE] Create a temporary file with name based on TEMPLATE and print its name. TEMPLATE must end with XXXXXX (e.g. [/dir/]nameXXXXXX). Without TEMPLATE, -t tmp.XXXXXX is assumed. -d Make directory, not file -q Fail silently on errors -t Prepend base directory name to TEMPLATE -p DIR Use DIR as a base directory (implies -t) -u Do not create anything; print a name Base directory is: -p DIR, else $TMPDIR, else /tmp * Updating TensorFlow to latest green. PiperOrigin-RevId: 209964879 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209976364 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209984345 * Updating TensorFlow to latest green. PiperOrigin-RevId: 209995175 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210006923 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210016839 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210026153 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210033704 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210039159 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210043179 * Directly import tensor.proto.h (the transitive import will be removed from tensor.h soon) We plan to remove the import variant.h from tensor.h; and variant.h brings in a lot of transitive imports (including protos like tensor.proto.h). To prepare, we're updating folks who this will break. PiperOrigin-RevId: 210043667 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210047695 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210055731 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210059511 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210064831 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210069954 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210075301 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210079614 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210083951 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210088160 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210105879 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210114079 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210124035 * Removed redundant std::string -> string conversions. PiperOrigin-RevId: 210127626 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210135188 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210142936 * Update bazel.rc PiperOrigin-RevId: 210148896 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210152249 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210162761 * Update implementation of optimization flags PiperOrigin-RevId: 210167619 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210172454 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210180873 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210187039 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210191416 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210194697 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210198019 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210208499 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210216276 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210224900 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210256024 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210258134 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210271455 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210322561 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210328210 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210350983 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210384280 * Directly import tensor.proto.h (the transitive import will be removed from tensor.h soon) We plan to remove the import variant.h from tensor.h; and variant.h brings in a lot of transitive imports (including protos like tensor.proto.h). To prepare, we're updating folks who this will break. PiperOrigin-RevId: 210392614 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210394881 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210406684 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210437609 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210448571 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210457512 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210464671 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210475410 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210480709 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210489481 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210493488 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210502178 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210513792 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210518475 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210548942 * Add new REST API to get status of models known to the ModelServer. The API is accessible via GET method on following URL: http://host:port/v1/models/${MODEL_NAME}[/versions/${MODEL_VERSION}] This closely mirrors `GetModelService.GetModelStatus` gRPC API. If no ${MODEL_VERSION} is specified, status for all versions of the model are returned. PiperOrigin-RevId: 210554222 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210558462 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210580715 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210613226 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210624649 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210634179 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210642123 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210648215 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210652892 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210657772 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210662340 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210667027 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210692176 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210697925 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210714741 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210730081 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210739779 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210751790 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210763533 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210785368 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210797415 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210808778 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210818742 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210827189 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210834551 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210840420 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210849745 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210854342 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210864386 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210871994 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210879339 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210890467 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210897025 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210906669 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210929579 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210946384 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210964552 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210977583 * Updating TensorFlow to latest green. PiperOrigin-RevId: 210999519 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211008596 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211015405 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211021634 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211025361 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211030060 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211034869 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211042744 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211047977 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211065383 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211079529 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211092679 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211101314 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211111559 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211122437 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211131001 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211140391 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211150256 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211159441 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211166367 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211173793 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211176461 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211179017 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211182228 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211185271 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211189621 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211192635 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211200782 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211209265 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211211166 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211231509 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211262324 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211294882 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211331648 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211351934 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211384717 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211394512 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211408480 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211434220 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211438961 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211467049 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211486592 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211499990 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211508655 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211520459 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211533217 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211541602 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211551681 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211559626 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211565612 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211569967 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211578437 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211582586 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211586976 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211593082 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211599422 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211604404 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211634227 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211641979 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211659890 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211668438 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211674601 * Removed a dependency on absl internal pkgs. PiperOrigin-RevId: 211681731 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211716173 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211726395 * Updating TensorFlow to latest green. PiperOrigin-RevId: 211732092

sathyarr · 2019-03-13T05:26:14Z

Thanks @netfs for the detailed explaination.

Actually we know about this limitation after reading the format of JSON requests. Since we separate batch data into items in instances array, it requires each item of instances has the same shape. Otherwise, passing optional parameters in different items is really confusing.

Because both the inputs and outputs of TensorFlow models are Tensor, have you considerated using the following format? It is much more compact which may improves performance and more similar with TensorFlow Serving gRPC APIs. We have another RESTful serving called simple_tensorflow_serving which uses this format and be compatible with all TensorFlow SavedModels.
{
  "instances": {
    "tag": ["foo", "bar"],
    "signal": [[1, 2, 3, 4, 5], [3, 4, 1, 2, 5]],
    "sensor": [[[1, 2], [3, 4]], [[4, 5], [6, 8]]]
  }
}

When I use columnar format to infer using TF Serving, I get the following error,

{
    "error": "seq_lens input must be 1-dim, not 2\n\t [[{{node model/att_seq2seq/encode/bidi_rnn_encoder/bidirectional_rnn/bw/ReverseSequence}}]]"
}

The model is based on seq2seq library.
Does the error mean the model does not support batching?

Also, the different format of input gives different error. Kindly see here

gautamvasudevan added bug stat:contributions welcome labels Aug 10, 2018

tensorflow-copybara closed this as completed in a5b7cfd Aug 21, 2018

peddybeats added the type:performance label Nov 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RESTful server checks consistency of batch size which breaks the model with arbitrary input or output shape #1047

RESTful server checks consistency of batch size which breaks the model with arbitrary input or output shape #1047

tobegit3hub commented Aug 10, 2018 •

edited

Loading

tobegit3hub commented Aug 10, 2018 •

edited

Loading

Uh oh!

netfs commented Aug 10, 2018

Uh oh!

tobegit3hub commented Aug 13, 2018 •

edited

Loading

Uh oh!

netfs commented Aug 14, 2018

Uh oh!

tobegit3hub commented Aug 15, 2018

Uh oh!

netfs commented Aug 15, 2018

Uh oh!

netfs commented Aug 15, 2018

Uh oh!

tobegit3hub commented Aug 20, 2018

Uh oh!

netfs commented Aug 20, 2018

Uh oh!

tobegit3hub commented Aug 24, 2018

Uh oh!

netfs commented Aug 24, 2018

Uh oh!

tobegit3hub commented Aug 27, 2018

Uh oh!

sathyarr commented Mar 13, 2019

Uh oh!

RESTful server checks consistency of batch size which breaks the model with arbitrary input or output shape #1047

RESTful server checks consistency of batch size which breaks the model with arbitrary input or output shape #1047

Comments

tobegit3hub commented Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

System information

Describe the problem

Exact Steps to Reproduce

Source code / logs

tobegit3hub commented Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netfs commented Aug 10, 2018

Uh oh!

tobegit3hub commented Aug 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netfs commented Aug 14, 2018

Uh oh!

tobegit3hub commented Aug 15, 2018

Uh oh!

netfs commented Aug 15, 2018

Uh oh!

netfs commented Aug 15, 2018

Uh oh!

tobegit3hub commented Aug 20, 2018

Uh oh!

netfs commented Aug 20, 2018

Uh oh!

tobegit3hub commented Aug 24, 2018

Uh oh!

netfs commented Aug 24, 2018

Uh oh!

tobegit3hub commented Aug 27, 2018

Uh oh!

sathyarr commented Mar 13, 2019

Uh oh!

tobegit3hub commented Aug 10, 2018 •

edited

Loading

tobegit3hub commented Aug 10, 2018 •

edited

Loading

tobegit3hub commented Aug 13, 2018 •

edited

Loading