NetSuite API call taking really long time

[huzaifaarain]

NetSuite API call taking too much time to respond, I did some research and found various solutions and tried almost all of them but no luck, I am not sure whether it's something due to my code or the NetSuite system itself. On average single API call is taking approx. 3 to 4 minutes, there are 1000 records in a single request.

SOAP Options for API Call

'soap_options'  => [
        "trace" => true,
        "keep_alive" => true,
        "connection_timeout" => 5000,
        "cache_wsdl" => WSDL_CACHE_NONE,
        "compression"   => SOAP_COMPRESSION_ACCEPT | SOAP_COMPRESSION_GZIP | SOAP_COMPRESSION_DEFLATE,
    ]

Code

try {
            $entityId = new SearchStringField;
            $entityId->operator = SearchStringFieldOperator::notEmpty;

            $search = new ContactSearchBasic;
            $search->entityId = $entityId;

            $request = new SearchRequest;
            $request->searchRecord = $search;

            $searchResponse = $this->service->search($request);
            if (!$searchResponse->searchResult->status->isSuccess) {
                $this->logger->error(
                    json_encode($searchResponse->searchResult->status->statusDetail),
                    [__FUNCTION__, self::class]
                );
                return false;
            }
            $this->logger->info(
                sprintf(
                    "Fetched %s records out of %s total records using page index %s",
                    count($searchResponse->searchResult->recordList->record),
                    $searchResponse->searchResult->totalRecords,
                    $searchResponse->searchResult->pageIndex
                ),
                [__FUNCTION__, self::class]
            );
            return $searchResponse->searchResult;
        } catch (\Throwable $th) {
            $this->logger->error($th->getMessage(), [__FUNCTION__, self::class]);
            throw $th;
        }

Any kind of suggestions and recommendations are appreciated.

[jrebs]

You say api calls are 3-4 minutes on average but your example only shows a search. As a first step to troubleshoot, have you tried a get() request to fetch a customer record by internalId? I'm curious if you're having this issue across all types of calls or if it's specific to your large result-set searches. Another thing maybe worth trying is experimenting with your soap options. For instance does disabling trace change the behaviour at all? Or removing the custom values like compression, etc, to see if any of them is a factor?

[huzaifaarain]

I wonder if there's a way to fetch all the records of the desired entity without going through a search. Like if I want to fetch all the contacts I shouldn't have to go through the ContactSearchBasic

[ezequiel9]

I 'been doing an integration of a quite big organization and I could not solve this problem. It is still really slow. I had to add background queues to prevent it in some cases. :(

In my case it is slow for contacts and Items, its doesnt take 3-4 minutes tho, just 10 seconds

[jaydenn7]

In my experience setting bodyFieldsOnly to false slowed the system down immensely, try disabling that if you don't need it

[MichaelBrenden]

We are also pulling lots of data from big tables for local storage. We ended up doing it via SuiteAnalyticsConnect/ODBC. We do the paging (1k records per batch) in our logic. We grab all the data we can, no 'make-lean' options enabled. Overall scheme now pulls around 50 rows / second max IIRC, fast enough so we haven't had to put it into a queue yet (BTW: we haven't yet noticed the fabled taffic-jam/connection-server-overload slowdowns yet). The ODBC connection setup still presents the biggest percentage of time consumption.

[neclimdul]

fwiw, unless something has changed the ODBC connector uses the same API so is subject to the same performance problems. But if it works for you great!

[incremental-altitude]

I think you need a more robust solution, that takes advantage of a suitescript to generate a CSV of that Saved Search. Then scan through the CSV with the php fopen function. Using phpToolkit to extract that much data in a repetitive soap search is laggy. Here's code to create the scheduled CSV dump to the Netsuite Filecabinet. (You'll just need the Saved Search ID from Netsuite and upload a blank CSV to a folder in the filecabinet and grab the Internal ID from that File. Then schedule to run once daily or whenever from your Netsuite Script deployment. Oh and make sure you set the Netsuite CSV file in the Filecabinet to public so you can grab the contents from a URL. /** * @NApiVersion 2.x * @NScriptType ScheduledScript * @NModuleScope SameAccount */ define(['N/task'], /** * @param {record} record * @param {search} search */ function(task) { var FILE_ID = 8498346; // Drop your placeholder blank csv file internal netsuite id here. var SEARCH_ID = 4932; // Drop your Saved Search internal netsuite id here. function execute(scriptContext) { var searchTask = task.create({ taskType: task.TaskType.SEARCH }); searchTask.savedSearchId = SEARCH_ID; searchTask.fileId = FILE_ID; var searchTaskId = searchTask.submit(); } return { execute: execute }; });// JavaScript Document ///// END SUITESCRIPT Next setup a php file with the following script. <? // BEGIN PHP CSV data extraction for use in program $file_source = " https://630303-sb3.app.netsuite.com/core/media/media.nl?id=8498346&c=630303_SB3&h=any8WhDBXrcnkBThUx8Z1n0IVmEnSr9dBoIXcMvRzVVaCAKt&_xt=.csv "; $header = NULL; $data = array(); if (($handle = fopen($file_source, "r")) !== FALSE) { while (($row = fgetcsv($handle, 1000, ',')) !== FALSE) { if(!$header) $header = $row; else $data[] = array_combine($header, $row); } fclose($handle); } /* echo '<pre>'; print_r($data); echo '</pre>'; */ $items_map = array(); $items_map = $data; // END ?> You can uncomment the <pre>print_r</pre> if you want to see what data the fopen script is extracting from your CSV. Can also load the data into an array or object or whatever you want to do with it once you get it out the CSV. This approach is way, way faster than constant API calls to Netsuite. Hope it helps and something to consider. Cheers.

…

On Thu, Jun 24, 2021 at 1:54 AM Jaden Shepherd ***@***.***> wrote: In my experience setting bodyFieldsOnly to false slowed the system down immensely, try disabling that if you don't need it — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#106 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPSLO2LSMFK7UXRSZBKKYLTULXEPANCNFSM47GCYRDA> .

-- Jarrod Dinwoodie

[incremental-altitude]

Almost forgot. You can set the f$file_source var to your public Netsuite CSV file URL $file_source = "";

…

On Thu, Jun 24, 2021 at 1:22 PM Jarrod ***@***.***> wrote: I think you need a more robust solution, that takes advantage of a suitescript to generate a CSV of that Saved Search. Then scan through the CSV with the php fopen function. Using phpToolkit to extract that much data in a repetitive soap search is laggy. Here's code to create the scheduled CSV dump to the Netsuite Filecabinet. (You'll just need the Saved Search ID from Netsuite and upload a blank CSV to a folder in the filecabinet and grab the Internal ID from that File. Then schedule to run once daily or whenever from your Netsuite Script deployment. Oh and make sure you set the Netsuite CSV file in the Filecabinet to public so you can grab the contents from a URL. /** * @NApiVersion 2.x * @NScriptType ScheduledScript * @NModuleScope SameAccount */ define(['N/task'], /** * @param {record} record * @param {search} search */ function(task) { var FILE_ID = 8498346; // Drop your placeholder blank csv file internal netsuite id here. var SEARCH_ID = 4932; // Drop your Saved Search internal netsuite id here. function execute(scriptContext) { var searchTask = task.create({ taskType: task.TaskType.SEARCH }); searchTask.savedSearchId = SEARCH_ID; searchTask.fileId = FILE_ID; var searchTaskId = searchTask.submit(); } return { execute: execute }; });// JavaScript Document ///// END SUITESCRIPT Next setup a php file with the following script. <? // BEGIN PHP CSV data extraction for use in program $file_source = " https://630303-sb3.app.netsuite.com/core/media/media.nl?id=8498346&c=630303_SB3&h=any8WhDBXrcnkBThUx8Z1n0IVmEnSr9dBoIXcMvRzVVaCAKt&_xt=.csv "; $header = NULL; $data = array(); if (($handle = fopen($file_source, "r")) !== FALSE) { while (($row = fgetcsv($handle, 1000, ',')) !== FALSE) { if(!$header) $header = $row; else $data[] = array_combine($header, $row); } fclose($handle); } /* echo '<pre>'; print_r($data); echo '</pre>'; */ $items_map = array(); $items_map = $data; // END ?> You can uncomment the <pre>print_r</pre> if you want to see what data the fopen script is extracting from your CSV. Can also load the data into an array or object or whatever you want to do with it once you get it out the CSV. This approach is way, way faster than constant API calls to Netsuite. Hope it helps and something to consider. Cheers. On Thu, Jun 24, 2021 at 1:54 AM Jaden Shepherd ***@***.***> wrote: > In my experience setting bodyFieldsOnly to false slowed the system down > immensely, try disabling that if you don't need it > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#106 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAPSLO2LSMFK7UXRSZBKKYLTULXEPANCNFSM47GCYRDA> > . > -- Jarrod Dinwoodie

-- Jarrod Dinwoodie

[MichaelBrenden]

2 years later almost to the day. Somebody mentioned ITEMS and CONTACTS being slow. Same here, via ODBC 2022_1 on Linux.

[neclimdul]

Its kinda hard for the library to do much about NetSuite being slow in a lot of places. I've just had to put all work happen in non-blocking queue. Especially if you're working with searches or listing of those items, its just slow which you can see even in the web UI.

[MichaelBrenden]

Understood. Did not mean to imply problem with this library. It's clearly NS/O going slowly. Item table particularly bad. We've just been pulling all data to local DB to do work much more quickly.