Arable API 2.2.0

Release Overview

Release Date: June 4, 2020

Instance Affected: Production

API: https://api.arable.cloud/api/v2/doc

3rd Party Sensors: Sentek Drill and Drop Soil Moisture Probe DD-MTS and DD-MT, Davis Wind Anemometer 6410 and 7911

Impact: High

Version: 2.2.0

While the Arable Web and Mobile had major releases this time around, Arable API 2.2.0 is a minor release. This is primarily because some of the major new features (Alert Subscription routes, GDD Growing Season routes, and Embedded BI Analytics dashboard routes) are currently only available to the Arable frontend while we let them mature before publishing in https://developer.arable.com/. 

Other improvements to the API include models for Leaf Wetness and ETc, as well as Mark status reliability and signal strength status.

New Features

• A new set of API routes for GDD and growing season tracking were implemented as part of this release. Users can now define a growing season (location, crop type, growth stages, & date range) and then retrieve GDD analytics specific to that growing season. As mentioned above, these new API routes will only be exposed to the Arable frontend in this release, and not yet in https://developer.arable.com/.
• In addition to the previously supported mobile push notifications, weather alerts are now available via SMS and voice calling. Users can now set organization-wide custom thresholds for alerts. These new API routes will only be exposed to the Arable frontend in this release and not yet in https://developer.arable.com/.
• The remote sensed weather API 2.0 (powered by IBM/The Weather Company) was extended to the API-user endpoint used by the Arable frontend. Both historical and forecasted weather can be accessed via this API. In this release, the historical data needs to be used with “and” in between measurements instead of “;”. This API is exposed in https://developer.arable.com/ only for usage with Arable Mark locations and is not replacing a generic weather service.
• A new Embedded BI capability was built around AWS QuickSight in the Arable backend and can now feed in Embedded iFrame in the Arable Web. Arable’s Customer Success team can help create and embed custom dashboards for customers on a per-project basis. The dashboards can be private to one organization or public to all organizations, but either way, the data is filtered down to just the data for the particular user.
• The leaf wetness calculation (LFW) was improved and now includes precipitation in addition to condensation. The improved LFW on the hourly table is a Leaf Wetness Prediction (LWP). LWP indicates whether the leaf is wet at any point within that hour, taking the value of either 1 for true or 0 for false. Subsequent to these changes, the true positive rate on the test set is at 73% and the true negative is 95%. The improved LFW primarily remains the same on the daily table. It is based on leaf wetness duration, representing the number of hours the leaf is wet on a given day, and calculated using the results of the hourly table. The only difference is that it is now expressed as an integer, due to the changes in the hourly table.
• As part of API 2.2.0, we performed an internal restructuring that relies on PostgresDB for API-device rather than using a parallel MongoDB instance. This will help streamline the backend data ingest procedure and will make it easier to troubleshoot and support issues. Note that over the next month, Arable will completely deprecate MongoDB.
• A new set of development and production environments are included with this release: Arable Labs Test (ALT), Staging (ALS), and Production (ALP). These environments provide better efficiency in feature testing and as such will allow a shorter deployment cycle.

Defect Fixes

• The implementation of the device status indication in the Arable backend has been refactored to allow for a more clean implementation. No change to the API endpoint was required for this refactoring.
• The reliability of the Mark signal strength status indicator has been improved so that it better aligns with what is experienced on the Mark’s local light UI. No change to the API endpoint was required for this modification.
• The ETc algorithm was modified to use the Dong net radiation in the Penman-Monteith formula, and is now independent of Mark longwave values. We apply the Mark version of Albedo in the Dong net radiation formula. Value limits are used to improve accuracy, including clipping Kc (crop coefficient) to 0 and clipping daily ET/ETc values to -1.
• Soil Moisture was previously provided as an average over the last 24 hours in the Summary endpoint that serves the map UI. In order to provide more actionable data, this is now changed to present Soil Moisture for the current hour instead.
• The response time for querying devices endpoint was closer to 800 seconds in the API 2.0. This has now been optimized to be max 400 msec (non-cached) and 200 msec (cached).
• An Internal Server Error message could occur for inactive and dormant devices when using the /data/hourly API. This has now been resolved.

Known Issues

• The current implementation to get the first hour of the weather forecast from IBM/The Weather Company is based on retrieving the remaining 15-minute forecast within the hour in real-time, as the current hour is not provided by the API on-demand. This means that we will not always have a full hour of data for temperature and precipitation based on API call timing. We plan to completely resolve this issue by caching the first hour forecast for all Arable Marks in an active state. This is planned for a future release. The current timing is as follows:

1. 00-07 min after the hour, we will get a full hour of data.
2. 08-22 min after the hour, we get 45 min of the hour’s data.
3. 23-37 min after the hour, we get 30 min of the hour’s data
4. 38-52 min after the hour, we get 15 min of the hour’s data
5. 53-59 min after the hour, we get 0 min of the hour’s data

• We have in some cases seen data gaps for specific devices based on humidity and temp sensor board response delays associated with ‘reset8s.’ While Firmware 1.1.0.9 will have increased the timeout and lessened the number of reset8s, we are continuing to watch for further improvements.
• It can take a long time to see measurement data after a new installation if the unit was not properly undeployed after the previous installation, as data will have been collected during transport and the Mark will send its data in FIFO mode.
• The Chlorophyll Index (CI) calculation is based on using the red-edge spectrometer readings. This has been shown in testing to give less accurate CI than if we use green and near infra-red instead. A modified CI algorithm will come in the next release.
• There is currently no way for customers to export all locations of a device ID, nor to delete a location via the API if the deployment is erroneous. This will be reviewed for a future release.
• If GPS isn't available in a deployment (e.g., deployed indoors), then the deployment may fail with a server-side exception. It is important to note that the Arable system is based on availability of GPS in order to provide accurate locations on maps and in calculations.
• The deployment of a Bridge is not immediately posted in the API (or Web/Mobile UI), even if the Mark or Bridge device light UI shows it as connected. In addition, if an external sensor and/or Bridge are disconnected, the device(s) will not be removed from the API (or Web/Mobile UI), as they are kept accessible to show historical data. A separation of data storage and sensor/Bridge status will be done in a future release.
• The summary endpoint that serves the Map UI in Arable Web & Mobile is today showing daily wind average. We will in a future release change this to instead show current hour wind data.
• While the API documentation at https://developer.arable.com/ is helpful, it is auto-generated and is missing a list of all available endpoints. We will add this into the Guide section in the future. In the meantime, if you need more help related to understanding the endpoints available on each API route, please contact Arable Customer Success.
• There is currently no way to know if the daily aggregations have a sufficient amount of hourly data to be considered useful for an external system and its algorithms in API 2.2.0. To solve this in upcoming releases, we will add a Data Quality Score flag to aggregations.
  There is currently no way to understand the number of hours within a day that have had more than 0.1mm of rain in API 2.2.0. This will be implemented in the next release as a precip hours endpoint.
• The remote-sensed wind in API 2.2.0 is at 10 meters height. In the next release, this will be lowered to 2 meters to be more aligned with the observed data from any Davis anemometer connected to the Arable Mark.