Optimize get_json_object Spark function using simdjson #5179

PHILO-HE · 2023-06-07T12:08:01Z

This PR proposes an implementation for Spark get_json_object function based on simdjson lib. This function returns a json object, represented by VARCHAR, from json string by searching user-specified path.

Spark source code link.

netlify · 2023-06-07T12:08:09Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`0c63bac`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/665eb3fb2155fc000889ae2a

rui-mo

Can we provide some background for this PR? Please also include Spark's implementation in the PR description.

rui-mo · 2023-08-31T07:41:45Z

velox/functions/sparksql/tests/JsonTest.cpp

+}
+
+} // namespace
+} // namespace facebook::velox::functions::sparksql::test


Add empty line at the end.

wanweiqiangintel · 2023-09-18T07:59:28Z

_ No description provided. _
@PHILO-HE, Could you please add some description to make the PR clear?

PHILO-HE · 2023-11-20T07:41:10Z

@rui-mo, please take a review. Thanks!

rui-mo · 2023-11-22T03:40:49Z

velox/functions/sparksql/SIMDJsonFunctions.h

+  FOLLY_ALWAYS_INLINE std::string getFormattedJsonPath(
+      const arg_type<Varchar>& jsonPath) {
+    // Makes a conversion from spark's json path, e.g.,
+    // converts "$.a.b" to "/a/b".


Can we move this comment above getFormattedJsonPath, and list all the rules for conversion? E.g. '$' will be ignored, '[' -> '/' etc.

rui-mo · 2023-11-22T03:43:12Z

velox/functions/sparksql/SIMDJsonFunctions.h

+        }
+      }
+      case ondemand::json_type::boolean: {
+        bool boolResult = false;


The init value seems to be not needed.

velox/functions/sparksql/SIMDJsonFunctions.h

rui-mo · 2023-11-22T03:45:18Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      }
+      case ondemand::json_type::boolean: {
+        bool boolResult = false;
+        rawResult.get_bool().get(boolResult);


Do we need to acquire the error code?

rui-mo · 2023-11-22T03:48:00Z

velox/functions/sparksql/SIMDJsonFunctions.h

+  // This is a simple validation by checking whether the obtained result is
+  // followed by valid char. Because ondemand parsing we are using ignores json
+  // format validation for characters following the current parsing position.
+  bool isValidEndingCharacter(const char* currentPos) {


Are we ensured all valid characters are covered?

At least we covered all valid characters in spark UT & customer workloads.

Is this Spark-specific? Do we not need same logic for Presto?

CC: @Yuhta

We just check document::at_end to make sure there is no trailing content except whitespace, that is enough for Presto and other apps in Meta. I think Spark could do the same.

rui-mo · 2023-11-22T03:48:49Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      } else {
+        rawResult =
+            ctx.jsonDoc.at_pointer(getFormattedJsonPath(jsonPath).data());
+      }


simdjson_resultondemand::value rawResult = formattedJsonPath_.has_value() ? one : the other

rui-mo · 2023-11-22T03:49:48Z

velox/functions/sparksql/tests/JsonFunctionsTest.cpp

+ protected:
+  std::optional<std::string> getJsonObject(
+      std::optional<std::string> json,
+      std::optional<std::string> jsonPath) {


rui-mo

Can we also update the documentation for get_json_object function?
https://github.com/facebookincubator/velox/blob/main/velox/docs/functions/spark/json.rst#json-functions-1

PHILO-HE · 2024-01-23T12:43:17Z

Hi @mbasmanova, could you spare some time to review this pr? Thanks!

mbasmanova · 2024-01-23T15:47:45Z

@PHILO-HE CI is red. Would you check?

mbasmanova

@PHILO-HE Is this function different from Presto? Curious, what are the differences?

mbasmanova

@PHILO-HE What's the motivation for this PR? Please, update PR description to clarify.

mbasmanova · 2024-01-23T15:49:46Z

velox/functions/sparksql/Register.cpp

@@ -124,7 +124,7 @@ void registerFunctions(const std::string& prefix) {
  // Register size functions
  registerSize(prefix + "size");

-  registerFunction<JsonExtractScalarFunction, Varchar, Varchar, Varchar>(


Are these any remaining usage of JsonExtractScalarFunction? If not, let's remove it.

I found it is still used by JsonExprBenchmark.cpp.

What is JsonExprBenchmark.cpp for? If it is just for benchmarking this function, then since function is no longer used, the benchmark is no longer needed and can be removed.

@mbasmanova, I found JsonExprBenchmark.cpp is used to benchmark a set of functions. Maybe, we can keep it in the source code?

@PHILO-HE We can keep the benchmark, but there is no point in benchmarking unused JsonExtractScalarFunction, hence, let's remove both the function and benchmark code.

Alternatively, we can move JsonExtractScalarFunction into the benchmark, assuming it is used to compare folly-based and simd-based implementations.

Just removed JsonExtractScalarFunction and the related usage in benchmark. Thanks!

velox/functions/sparksql/SIMDJsonFunctions.h

mbasmanova · 2024-01-23T15:51:13Z

velox/functions/sparksql/SIMDJsonFunctions.h

+template <typename T>
+struct SIMDGetJsonObjectFunction {
+  VELOX_DEFINE_FUNCTION_TYPES(T);
+  std::optional<std::string> formattedJsonPath_;


Make this variable private.

mbasmanova · 2024-01-23T15:51:29Z

velox/functions/sparksql/SIMDJsonFunctions.h

+
+  // Makes a conversion from spark's json path, e.g., converts
+  // "$.a.b" to "/a/b".
+  FOLLY_ALWAYS_INLINE std::string getFormattedJsonPath(


Make this private. Do you really need to inline this method?

Just made it private.
This function can be frequently called for handling non-constant input path. May be better to make it inlined.

mbasmanova · 2024-01-23T15:51:42Z

velox/functions/sparksql/SIMDJsonFunctions.h

+    }
+  }
+
+  FOLLY_ALWAYS_INLINE simdjson::error_code extractStringResult(


private; ditto other places

Yuhta · 2024-01-23T16:18:38Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      if (error) {
+        return false;
+      }
+    } catch (simdjson_error& e) {


We should not have any exception here, every error should be represented in the return code

Thanks for your comment!
I found simdjson lib can throw exceptions for some cases.
For example, if we don't keep try/catch here, the below test will fail with an exception thrown:
"The JSON document has an improper structure: missing or superfluous commas, braces, missing key"

EXPECT_EQ(getJsonObject(R"({"hello"-3.5})", "$.hello"), std::nullopt);

@Yuhta Jimmy, I wonder if that's why we are seeing unhandled "Unexpected trailing content in the JSON input" errors in some queries: T175957555 . Since throwing exceptions is expensive (under TRY), I wonder if there is a way to tell simdjson to not throw but rather return error code. Maybe we can open an issue in simdjson GitHub repo and ask about that.

@PHILO-HE @mbasmanova Certain simdjson functions are throwing, you can tell from the signature, but all throwing functions have an alternative version that is returning error code. So we need to find out which function is throwing and replace them, then we can remove the catch here.

@Yuhta, thanks for your suggestion! I found we can directly check the returned error code. The try/catch was just removed.

PHILO-HE · 2024-01-26T16:19:13Z

@PHILO-HE What's the motivation for this PR? Please, update PR description to clarify.

Hi @mbasmanova, I just updated PR description and fixed the comments. Thanks for your review!

PHILO-HE · 2024-01-29T13:17:16Z

@mbasmanova, could you take a review further? Thanks!

mbasmanova

@PHILO-HE

Add simdjson based get_json_object Spark function

The PR title suggests that this PR adds a new Spark function. However, the description suggests that the changes are to optimize an existing function not to add a new function. Which is correct? It would be nice to align PR title, PR description, and code changes.

mbasmanova · 2024-01-29T23:57:19Z

velox/docs/functions/spark/json.rst

@@ -22,6 +22,8 @@ JSON Functions

 .. spark:function:: get_json_object(json, path) -> varchar

-    Extracts a json object from path::
+    Extracts a json object from ``path``. Returns NULL if it finds json string


It would be nice to document what values for 'path' are supported? Maybe provide a link to some existing documentation for path?

@mbasmanova, I didn't find official document for valid json path. Just documented the pattern and gave some examples.

mbasmanova · 2024-01-29T23:58:35Z

velox/docs/functions/spark/json.rst

+        SELECT get_json_object('{"a":"b"}', '$.a'); -- 'b'
+        SELECT get_json_object('{"a":{"b":"c"}}', '$.a'); -- '{"b":"c"}'


Does this function take an argument of type VARCHAR and return result of type JSON? (this is what signature on L23 suggests)? Or, does this function return VARCHAR result?

All inputs are VARCHAR and output is also VARCHAR. Just clarified in the doc. Thanks!

mbasmanova · 2024-01-30T14:26:21Z

velox/docs/functions/spark/json.rst


-    Extracts a json object from path::
+    Returns a json object, represented by VARCHAR, from ``jsonString`` by searching ``path``.
+    Valid ``path`` should contain "[index]", "[field]" or ".field" to define a JSON path.


It is necessary for 'path' to start with $? If so, would you clarify that here? Do you have a pointer to Spark code that parses 'path'?

Yes, Spark requires $ is the initial character in path. Just updated the doc.
Here is spark code, which starts the parsing from "$".
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L94

mbasmanova · 2024-01-30T14:28:33Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      const arg_type<Varchar>& jsonPath) {
+    char formattedJsonPath[jsonPath.size() + 1];
+    int j = 0;
+    for (int i = 0; i < jsonPath.size(); i++) {


This logic is pretty important as it effectively defines what paths are supported. It would be nice to extract it into a separate header with clear documentation and dedicated unit tests.

For example, looks like we allow $ to appear anywhere within the path. Is this correct? Can you provide an example of a valid 'path' where $ appears somewhere in the middle?

looks like we allow $ to appear anywhere within the path. Is this correct?

Just found this code is wrong. $ should be in the initial position and should not occur then. I verified this behavior with Spark query. Just updated this code and added a test case.

PHILO-HE · 2024-03-11T13:18:12Z

Hi @mbasmanova, could you review this pr further?

As discussed before, getJsonPointerPath is important since it converts json path to json pointer path used for searching, e.g, from "$.a.b[0]" to "/a/b/0". I recently found such conversion has already been implemented in simdjson lib, but only supported from 3.8.0 (the version is 3.2.0 in velox by now). So I copied those code from simdjson lib for path conversion.

mbasmanova · 2024-03-11T13:22:25Z

I recently found such conversion has already been implemented in simdjson lib, but only supported from 3.8.0 (the version is 3.2.0 in velox by now). So I copied those code from simdjson lib for path conversion.

Let's upgrade to simdjson 3.8.0.

PHILO-HE · 2024-04-16T13:32:53Z

I recently found such conversion has already been implemented in simdjson lib, but only supported from 3.8.0 (the version is 3.2.0 in velox by now). So I copied those code from simdjson lib for path conversion.

Let's upgrade to simdjson 3.8.0.

Hi @mbasmanova, as simdjson has been upgraded, I just updated this pr to use its new API. Could you please review again? Thanks!

PHILO-HE · 2024-04-23T15:31:34Z

@mbasmanova, could you review this pr again? Thanks!

mbasmanova

@PHILO-HE Some comments. Please, rebase.

mbasmanova · 2024-04-24T14:36:39Z

velox/functions/sparksql/SIMDJsonFunctions.h

+
+#include "velox/functions/prestosql/SIMDJsonFunctions.h"
+
+using namespace simdjson;


no 'using namespace' in header files, please

mbasmanova · 2024-04-24T14:37:16Z

velox/functions/sparksql/SIMDJsonFunctions.h

+namespace facebook::velox::functions::sparksql {
+
+template <typename T>
+struct SIMDGetJsonObjectFunction {


Let's remove 'SIMD' prefix. Presto functions have it now, because, originally there were 2 sets of functions, but we should rename these as well.

mbasmanova · 2024-04-24T14:40:24Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      }
+      pairEnd = result.find("]", pairBegin);
+      if (pairEnd == std::string::npos || result[pairEnd - 1] != '\'') {
+        return "-1";


This seems a bit hacky. Are you relying on simdjson to reject this path later? It would be cleaner to raise an exception here and provide a clear error message.

This function seems to be working around a limitation on simdjson. Have you already asked in simdjson project if that limitation can be removed? Anyway, please, update PR description to explain this limitation and how you are working around it.

Are you relying on simdjson to reject this path later? It would be cleaner to raise an exception here and provide a clear error message.

@mbasmanova, yes, we depend on simdjson to return error code for illegal path, then this function will return NULL result at last, instead of throwing exception.

Have you already asked in simdjson project if that limitation can be removed?

I just left a comment in simdjson community to discuss this limitation:
simdjson/simdjson#2070 (comment)

@PHILO-HE

The API that returns NULL when json or path are invalid is challenging. Throwing an error would be better.

Note that simdjson has limited support for JSONPath, hence, it rejects both invalid and valid-but-not-supported paths. If Spark supports full JSONPath spec or a wider subset of the spec than simdjson, then this implementation will produce incorrect results.

Hence, some questions.

What subset of JSONPath is supported in Spark?

Is there any particular reason to not use SIMDJsonExtract[Scalar]Function directly or implement this in a similar way by leveraging velox/functions/prestosql/json/SIMDJsonExtractor.h ?

Does Spark validate JSON document fully and return null if it is not valid? simdjson doesn't do that and may succeed at extracting the value from an invalid document. This will then cause Gluten to produce wrong results.

@PHILO-HE

The API that returns NULL when json or path are invalid is challenging. Throwing an error would be better.

Note that simdjson has limited support for JSONPath, hence, it rejects both invalid and valid-but-not-supported paths. If Spark supports full JSONPath spec or a wider subset of the spec than simdjson, then this implementation will produce incorrect results.

Hence, some questions.

What subset of JSONPath is supported in Spark?

@mbasmanova, thanks for your comment!
Spark has its own code to parse Json path. I have addressed all inconsistency found in Spark tests. For example, ['name']['id'] is supported by Spark, but not by Simdjson. This pr fixed this inconsistency by just pre-processing the path. So I feel it's better to return NULL instead of throwing exception for invalid path.

Is there any particular reason to not use SIMDJsonExtract[Scalar]Function directly or implement this in a similar way by leveraging velox/functions/prestosql/json/SIMDJsonExtractor.h ?

The proposed pr has some handlings to align with Spark, e.g., check the validity of JSON document. And with this proposed patch, all Spark test cases can pass.

Does Spark validate JSON document fully and return null if it is not valid? simdjson doesn't do that and may succeed at extracting the value from an invalid document. This will then cause Gluten to produce wrong results.

Yes, Spark fully validates JSON document. Simdjson only checks the validity before the position of extracted result. That's why this pr checked the characters after this position. Though it's not a full validation, it can meet our Spark users' requirement.

@PHILO-HE

Spark has its own code to parse Json path.

Do you have a pointer? Which subset (or superset) of JsonPath is supported in Spark?

I have addressed all inconsistency found in Spark tests.

It is good to have all tests pass, but it doesn't guarantee the the semantics match 100%. How do you know that tests cover all cases? We may need to read Spark's code for handing JsonPath to understand what it supports and how.

Also, it seems that simdjson would be parsing JsonPath repeatedly for every row. Is this desired? If we need to parse JsonPath ourselves anyway, why not handle that path as well?

mbasmanova · 2024-04-24T14:40:35Z

velox/functions/sparksql/SIMDJsonFunctions.h

+  // Spark's json path requires field name surrounded by single quotes if it is
+  // specified in "[]". But simdjson lib requires not. This method just removes
+  // such single quotes, e.g., converts "['a']['b']" to "[a][b]".
+  FOLLY_ALWAYS_INLINE std::string removeSingleQuotes(


no need for FOLLY_ALWAYS_INLINE here; please, remove

mbasmanova · 2024-04-24T14:41:53Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      const arg_type<Varchar>& json,
+      const arg_type<Varchar>& jsonPath) {
+    // Spark requires the first char in jsonPath is '$'.
+    if (jsonPath.size() < 1 || jsonPath.data()[0] != '$') {


this check is repeated in L36; perhaps, add a small helper function to avoid copy-paste

mbasmanova · 2024-04-24T14:48:43Z

velox/functions/sparksql/SIMDJsonFunctions.h

+  FOLLY_ALWAYS_INLINE simdjson::error_code extractStringResult(
+      simdjson_result<ondemand::value> rawResult,
+      out_type<Varchar>& result) {
+    simdjson::error_code error;


this variable is not needed; this function can simply return a boolean

mbasmanova · 2024-04-24T14:49:15Z

velox/functions/sparksql/SIMDJsonFunctions.h

+      // can make simdjson's internal parsing position moved and then we
+      // can check the validity of ending character.
+      case ondemand::json_type::number: {
+        switch (rawResult.get_number_type()) {


I wonder if this code repeats somewhere else. Would you check?

Just checked again. I didn't find the repeat code anywhere else.

mbasmanova · 2024-04-24T14:49:40Z

velox/functions/sparksql/tests/JsonFunctionsTest.cpp

+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "velox/functions/prestosql/types/JsonType.h"


Why is this include? We shouldn't use Presto types in Spark code.

mbasmanova · 2024-04-24T14:50:10Z

velox/functions/sparksql/tests/JsonFunctionsTest.cpp

+          "$[1].other[1]"),
+      "v2");
+
+  // Field not found.


This test is rather long. Consider extracting invalid cases into a separate test method.

mbasmanova · 2024-04-24T14:51:04Z

velox/functions/sparksql/tests/JsonFunctionsTest.cpp

+class JsonFunctionTest : public SparkFunctionBaseTest {
+ protected:
+  std::optional<std::string> getJsonObject(
+      const std::optional<std::string>& json,


No need to use optional here as this method doesn't allow unset inputs.

PHILO-HE · 2024-04-26T16:48:38Z

velox/functions/prestosql/benchmarks/JsonExprBenchmark.cpp

@@ -189,8 +189,6 @@ class JsonBenchmark : public velox::functions::test::FunctionBenchmarkBase {
        {"folly_json_array_length"});
    registerFunction<SIMDJsonArrayLengthFunction, int64_t, Json>(
        {"simd_json_array_length"});
-    registerFunction<JsonExtractScalarFunction, Varchar, Json, Varchar>(


@mbasmanova, I note you have moved JsonExtractScalarFunction to this file. Do I need to put the deleted code back for this benchmark test?

Yes, I think changes to this file can be reverted. Thanks.

@mbasmanova, just reverted. Thanks!

mbasmanova · 2024-05-08T12:34:21Z

velox/functions/sparksql/JsonFunctions.h

+      }
+      pairEnd = result.find("]", pairBegin);
+      if (pairEnd == std::string::npos || result[pairEnd - 1] != '\'') {
+        return "-1";


Let's not pass knowngly invalid path to simdjson, but instead cut processing short and return NULL.

mbasmanova · 2024-05-08T15:34:06Z

I found some info about get_json_path implementation in Spark.

https://issues.apache.org/jira/browse/SPARK-37857

It seems the first implementation of Spark get_json_object does replicate Hive behaviour (https://github.com/apache/spark/pull/7901)

In hive documentation (https://cwiki.apache.org/confluence/display/hive/languagemanual+udf) it is stated that recursive decent (i.e ".." notation) is not supported

A limited version of JSONPath is supported:

$ : Root object
. : Child operator
[] : Subscript operator for array
* : Wildcard for []

Syntax not supported that's worth noticing:

: Zero length string as key
.. : Recursive descent
@ : Current object/element
() : Script expression
?() : Filter (script) expression.
[,] : Union operator
[start:end.step] : array slice operator

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 7, 2023

rui-mo reviewed Aug 31, 2023

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch 2 times, most recently from 8ec0ed4 to ebab33c Compare November 15, 2023 08:00

PHILO-HE changed the title ~~Support get_json_object function for spark based on simdjson lib~~ Add simdjson based get_json_object Spark function Nov 15, 2023

PHILO-HE marked this pull request as ready for review November 15, 2023 08:13

zhouyuan mentioned this pull request Nov 16, 2023

[VL] Potential compatibility issues for Spark get_json_object function apache/incubator-gluten#3723

Open

PHILO-HE force-pushed the get-json-object-upstream branch from 8ef9856 to 27ae51d Compare November 16, 2023 14:06

rui-mo reviewed Nov 22, 2023

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch from 00c9c0a to 65fcace Compare November 24, 2023 06:59

PHILO-HE force-pushed the get-json-object-upstream branch from 65fcace to f7d83cd Compare January 15, 2024 00:52

PHILO-HE force-pushed the get-json-object-upstream branch from f7d83cd to 6da58d6 Compare January 23, 2024 12:41

mbasmanova requested a review from Yuhta January 23, 2024 15:47

mbasmanova reviewed Jan 23, 2024

View reviewed changes

Yuhta reviewed Jan 23, 2024

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch from 6da58d6 to b4e49e9 Compare January 26, 2024 16:08

mbasmanova reviewed Jan 29, 2024

View reviewed changes

mbasmanova changed the title ~~Add simdjson based get_json_object Spark function~~ Optimize get_json_object Spark function using simdjson Jan 30, 2024

mbasmanova reviewed Jan 30, 2024

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch 3 times, most recently from 1e804eb to 8eaa089 Compare March 11, 2024 09:50

PHILO-HE mentioned this pull request Mar 12, 2024

Upgrade simdjson lib to 3.7.0 (from 3.2.0) #9041

Closed

PHILO-HE force-pushed the get-json-object-upstream branch from 0a65b10 to 217c90f Compare April 16, 2024 12:58

This was referenced Apr 23, 2024

Fold JsonFunctions.h into JsonExprBenchmark.cpp #9575

Closed

[VL] Json support in Gluten/Velox tracker apache/incubator-gluten#5504

Open

mbasmanova reviewed Apr 24, 2024

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch from 9d9ffbe to 2d3c983 Compare April 26, 2024 10:20

PHILO-HE commented Apr 26, 2024

View reviewed changes

PHILO-HE force-pushed the get-json-object-upstream branch 2 times, most recently from 7d4685c to bd51642 Compare May 6, 2024 14:05

mbasmanova reviewed May 8, 2024

View reviewed changes

PHILO-HE added 2 commits June 4, 2024 14:27

Initial commit

7e51834

Fix compile error

0c63bac

PHILO-HE force-pushed the get-json-object-upstream branch from bd51642 to 0c63bac Compare June 4, 2024 06:28

		SELECT get_json_object('{"a":"b"}', '$.a'); -- 'b'
		SELECT get_json_object('{"a":{"b":"c"}}', '$.a'); -- '{"b":"c"}'


		#include "velox/functions/prestosql/SIMDJsonFunctions.h"

		using namespace simdjson;

Optimize get_json_object Spark function using simdjson #5179

Are you sure you want to change the base?

Optimize get_json_object Spark function using simdjson #5179

Conversation

PHILO-HE commented Jun 7, 2023 • edited

netlify bot commented Jun 7, 2023 • edited

✅ Deploy Preview for meta-velox canceled.

rui-mo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanweiqiangintel commented Sep 18, 2023

PHILO-HE commented Nov 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yuhta Jan 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rui-mo left a comment

Choose a reason for hiding this comment

PHILO-HE commented Jan 23, 2024

mbasmanova commented Jan 23, 2024

mbasmanova left a comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yuhta Jan 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE commented Jan 26, 2024

PHILO-HE commented Jan 29, 2024

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE Mar 6, 2024 • edited

Choose a reason for hiding this comment

PHILO-HE commented Mar 11, 2024

mbasmanova commented Mar 11, 2024

PHILO-HE commented Apr 16, 2024

PHILO-HE commented Apr 23, 2024

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE commented Jun 7, 2023 •

edited

netlify bot commented Jun 7, 2023 •

edited

Yuhta Jan 30, 2024 •

edited

Yuhta Jan 26, 2024 •

edited

PHILO-HE Mar 6, 2024 •

edited

PHILO-HE Apr 26, 2024 •

edited