The data people learning platform is online: www.shujurenclub.com

About the author

@muyan

Data Product Manager;

Responsible for intelligent labels, index management, algorithm platform, data service platform, data analysis platform, data buried point collection and other data products

Member of the “Federation of Data People Creators”.

 

Label scoring is an important measure of label governance, by scoring labels, you can clearly and intuitively evaluate labels from various dimensions, grasp the real use of labels, and continuously optimize labels to help business operations. At the same time, it can also help the data team determine which tags should be more invested in computing and storage resources, and reasonably plan cluster resources.

First, why use tag scoring

After the early label system design and label processing, the label can finally be put online, so that business personnel can use it and play a valuable role!

With the tag online for a period of time, we began to care about the daily occupation of computing resources and storage space, running out of hundreds of tags, how much business students really used, business revenue can cover the cost of data? After the label is launched, how is its quality, and is there a situation where the old rules do not apply and need continuous optimization?

With this problem in mind, we need a way to evaluate the use of tags after they go live and identify the value of each tag. Referring to the forms of movie ratings, Huabei ratings, etc., we decided to also give the labels a score and an order, which is simple and clear.

Second, the label scoring model

Label scoring model, after consideration we selected 5 dimensions as scoring parameters:

Total label score = a * Label usage score + b * Label attention score + c * Label quality score + d * Label continuous optimization score + e * Label safety score

Among them, the degree of label use, label attention, label quality, and continuous optimization of labels are the core dimensions, and the label safety can be considered according to the actual situation. a, b, c, d, e are weights, which add up to 100%.

1. Label usage scoring

Label usage to evaluate the use of labels being analyzed and external systems.

In the Kangaroo Cloud Label product, the label has several use cases:

(1) Label reference: such as atomic tags are applied by derivative tags, derivative tags are referenced by combined tags, etc., based on this scenario, calculate the “number of tag citations” indicator.

(2) Label analysis: The “number of label analysis” indicators is calculated when the label is analyzed in the image analysis functions such as label circle group, group portrait, group comparison, and significance analysis.

(3) Label call: The number of times the label is queried by external applications through the data API, and the “number of label calls” indicator is calculated.

Based on the above three indicators, we first use the Sigmoid function to convert the indicators into scores, and then summarize the scores of each indicator into a label usage score.

2. Tag attention score

Tag attention to evaluate how much it is searched, viewed, and collected.

In kangaroo cloud labeling products, label attention is related to the following scenarios:

(1) Label search: When a label is searched by a user in the label market, the “number of label searches” indicator is calculated.

(2) Label viewing: The number of times the label is clicked to view basic information, analysis pages, etc., and the “number of label views” is calculated.

(3) Label collection: The number of users who collect the label is calculated.

The above 3 indicators can reflect the popularity of label attention, we still use the Sigmoid function to convert the indicator into a score, and then the score of each indicator is weighted into a label attention score.

3. Label quality score

Label quality, to evaluate the user’s marking situation, reflecting the rationality of the labeling rules.

When we define the label and the label value, after calculation, the label value is very small on the user, which means that our rule implementation is not reasonable. For example, we define the label of “activity”, which is divided into “high activity, medium activity, low activity”, etc., but the real users who are marked with this label are less than 70%, and a large part of the proportion is null, and the label is not labeled, indicating that the label value rules we have formulated have loopholes and need to be improved.

The system calculates the “label coverage” of each label, normalizes the coverage to a score, and converts it into a score.

4. Continuous optimisation scoring

Continuous optimization to evaluate whether to optimize the tag after it is online.

In the life cycle of customers, there are constantly new users flowing in and silent users losing; Company strategy adjustments, product releases, etc. will affect customer behavior, these changes we need to present in the form of data, so we need to constantly adjust our labeling strategy according to business adjustments, customer changes, in order to pursue through the label directly and quickly reflect the customer situation, guide business operations.

Continuous optimization, which we evaluate through the “Number of Label Optimizations” metric, refers to the number of times a label has been edited and republished after it has gone live. We also use the Sigmoid function to convert the indicator into a score.

5. Security scoring

Label safety does not reflect the popularity of the label, but it is also used as a dimension of the label score, which can be considered according to the situation of the enterprise.

In Kangaroo Cloud Label products, tag security-related policies are:

(1) Visibility of the label: The user range of the label can be edited and viewed.

(2) Whether the use of the label needs to apply for authorization: After the label is released, others use the label, and whether they need to apply for approval.

(3) Whether the label is row-level permission control: Above we control the column permissions of the label, and the row-level permissions reflect whether the label has row-level permissions set.

(4) Whether the label is desensitized: whether the label is desensitized.

Based on the security policy configuration of the label, we also use the scoring method to evaluate.

Based on the above 5 dimensions of the score, we summarize the weighted summary according to the previously mentioned formula to obtain the total score.

Third, the application of label scoring

Based on tag scoring, in order to more intuitively let tag administrators and business personnel view popular tags, silent tags, etc., it is presented through the way of leaderboards.

1. Top tab leaderboard

Based on the three angles of label usage, attention, and continuous optimization, the popular score of the label is calculated, and the popular label of TOP n is displayed.

Popular labels indicate that these labels are often used, and we need to continue to pay attention to the normal operation and quality of these labels to ensure business use.

2. Silent tab leaderboard

The reverse ranking of popular tags is silent tags, which indicate that the usage of these tags is very low, and you can consider going offline regularly to save cluster resources.

3. Comprehensive leaderboards

The comprehensive leaderboard is sorted according to the comprehensive score of the label, and the label is evaluated from several dimensions such as label usage, attention, continuous optimization, quality, and safety.

4. Ranking in the sub-list of label usage, attention, continuous optimization, quality and safety

Users can view the leaderboards of each sub-dimension of label usage, attention, continuous optimization, quality, and safety according to the dimensions they are more concerned about. At the same time, you can view the specific metrics of each label, such as the usage dimension, you can view the current number of references, analysis times, and calls of each label, and analyze the specific indicators to meet different label analysis scenarios.

After the label scoring model is launched, we need to adjust the weights of different dimensions according to the actual situation to meet our own actual conditions. When after a period of application, everyone recognizes this set of evaluation logic, you can transform the static scoring display into dynamic alarms, automated governance, etc., and set label quality alarms and scoring alarms to automatically notify label administrators and responsible persons.

The above is the scoring logic applied in the product, hoping to help everyone, and can also put forward different ideas to optimize the scoring model to achieve a better label governance effect.

If you want to know more about data knowledge, please also welcome to see the book “Big Data Practice Road: Data Middle Office + Data Analysis + Product Application” written by 7 big factory data products.