System and method for determining damage on crops

{
  "publication_number": "US2023017425A1",
  "country": "US",
  "kind": "A1",
  "title": "System and method for determining damage on crops",
  "abstract": "A computer-implemented method, computer program product and computer system (100) for determining the impact of herbicides on crop plants (11) in an agricultural field (10). The system includes an interface (110) to receive an image (20) with at least one crop plant representing a real world situation in the agricultural field (10) after herbicide application. An image pre-processing module (120) rescales the received image (20) to a rescaled image (20 a) matching the size of an input layer of a first fully convolutional neural network (CNN 1) referred to as the first CNN. The first CNN is trained to segment the rescaled image (20 a) into crop (11) and non-crop (12, 13) portions, and provides a first segmented output (20 s 1) indicating the crop portions (20 c) of the rescaled image with pixels belonging to representations of crop. A second fully convolutional neural network (CNN 2), referred to as the second CNN, is trained to segment said crop portions into a second segmented output (20 s 2) with one or more sub-portions (20 n, 20 l) with each sub-portion including pixels associated with damaged parts of the crop plant showing a respective damage type (11 - 1, 11 - 2). A damage measurement module (130) determines a damage measure (131) for the at least one crop plant for each damage type (11 - 1, 11 - 2) based on the respective sub-portions of the second segmented output (20 s 2) in relation to the crop portion of the first segmented output (20 s 1).",
  "claims": [
    "1. A computer-implemented method (1000) for determining damage on crop plants (11) after herbicide application in an agricultural field (10), comprising: receiving (1100) an image (20) representing a real world situation in the agricultural field (10) after herbicide application, with at least one crop plant; rescaling (1200) the received image (20) to a rescaled image (20 a) matching the size of an input layer of a first convolutional neural network (CNN 1) referred to as the first CNN, the first CNN (CNN 1) being trained to segment the rescaled image (20 a) into crop (11) and non-crop (12, 13) portions by using color transformation processes in a data augmentation stage allowing the first CNN to learn to distinguish between soil related pixels and necrotic crop related pixels, and to provide a first segmented output as a mask identifying the crop portions in the rescaled image including necrotic parts of the crop plant; applying (1300) the first CNN (CNN 1) to the rescaled image (20 a) to provide, to a second convolutional neural network (CNN 2) referred to as the second CNN, the first segmented output (20 s 1), the second CNN (CNN 2) being a semantic segmentation neural network trained to segment said crop portions into one or more sub-portions (20 n, 20 l) with each sub-portion including pixels associated with damaged parts of the crop plant showing a respective damage type (11 - 1, 11 - 2) being a particular damage type of a plurality of damage types comprising necrosis and at least one further damage type; applying (1400) the second CNN (CNN 2) to the crop portions (20 c) of the rescaled image to identify, in a second segmented output (20 s 2), damaged parts of the at least one crop plant by damage type (11 - 1, 11 - 2) for the plurality of damage types; and determining (1500) a damage measure (131) for the at least one crop plant for each damage type (11 - 1, 11 - 2) based on the respective sub-portions of the second segmented output (20 s 2) in relation to the crop portion of the first segmented output (20 s 1). 2. The method of claim 1, wherein the types of damage further comprise any of leaf curling and bleaching. 3. The method of claim 1, wherein the first CNN and/or the second CNN is based on a segmentation topology selected from the group of: Fully Convolutional Dense Net, UNet, and PSPNet. 4. The method of claim 1, wherein the first CNN is trained using a first loss function (LF 1) to measure the performance of the first CNN to segment the resealed image (20 a) into crop (11) and non-crop portions with at least a first non-crop portion associated with soil (12) and a second non-crop portion associated with non-crop green plants (13). 5. The method of claim 1, wherein the second CNN is trained using a second loss function (LF 2) selected from the group of: mean squared error loss, dice loss, generalized dice loss, focal loss, or Tversky loss. 6. The method of claim 1, wherein the resealed image (20 a) is reduced in size compared to the received image (20) while the damage symptoms associated with any type of damage (11 - 1, 11 - 2) are still visible on the resealed image. 7. The method of claim 6, wherein a training data set for training the first CNN includes: images with healthy crop plants, images with damaged crop plants with damages of different damage types, and images with damaged or healthy crop plants and non-crop plants. 8. The method of claim 7, wherein a further training data set for training the second CNN includes images with damaged crop plants with damages of different damage types. 9. The method of claim 7, wherein a particular subset of images the training data set is augmented by transforming the images of the subset from the RGB color space to another color space; modifying intensity values of respective transformed color channels randomly; and transforming the modified images back into the RGB color space. 10. A non-transitory computer-readable medium having instructions thereon encoding a computer program product for determining the impact of herbicides on crop plants (11) in an agricultural field (10), wherein the instructions, when executed by memory of a computing device and executed by at least one processor of a computing device, cause the at least one processor to execute the steps of the computer-implemented method according to claim 1. 11. A computer system (100) for determining damage on crop plants (11) after herbicide application in an agricultural field (10), comprising: an interface (110) configured to receive an image (20) representing a real world situation in the agricultural field (10) after herbicide application, with at least one crop plant; an image pre-processing module (120) configured to rescale the received image (20) to a resealed image (20 a) matching the size of an input layer of a first convolutional neural network (CNN 1) referred to as the first CNN; the first CNN, being trained to segment the resealed image (20 a) into crop (11) and non-crop (12, 13) portions by using color transformation processes in a data augmentation stage allowing the first CNN to learn to distinguish between soil related pixels and necrotic crop related pixels, and to provide a first segmented output as a mask identifying the crop portions in the resealed image including necrotic parts of the crop plant; a second convolutional neural network (CNN 2), referred to as the second CNN, being a semantic segmentation neural network trained to segment said crop portions into a second segmented output (20 s 2) with one or more sub-portions (20 n, 20 l) with each sub-portion including pixels associated with damaged parts of the crop plant showing a respective damage type (11 - 1, 11 - 2) being a particular damage type of a plurality of damage types comprising necrosis and at least one further damage type; a damage measurement module (130) configured to determine a damage measure (131) for the at least one crop plant for each damage type (11 - 1, 11 - 2) based on the respective sub-portions of the second segmented output (20 s 2) in relation to the crop portion of the first segmented output (20 s 1). 12. The system of claim 11, wherein the damage types further comprise any of leaf curling and bleaching. 13. The system of any of claim 11, wherein the first CNN and/or the second CNN is based on a segmentation topology selected from the group of: Fully Convolutional Dense Net, UNet, and PSPNet. 14. The system of any of claim 11, wherein the first CNN is trained using a first loss function (LF 1) to measure the performance of the first CNN to segment the rescaled image (20 a) into crop (11) and non-crop (12, 13) portions, the second CNN is trained using a second loss function (LF 2) selected from the group of: mean squared error loss, dice loss, generalized dice loss, focal loss, or Tversky loss. 15. A computer system (100 ′) for determining biomass reduction of crop plants (11) after herbicide application in an agricultural field, comprising: an interface (110) configured to receive a test image (20) representing a real world situation of a test plot (10 - 1) in the agricultural field after herbicide application, with at least one crop plant; an image pre-processing module (120) configured to rescale the received image (20) to a rescaled image (20 a) matching the size of an input layer of a convolutional neural network (CNN 1) referred to as CNN; the CNN, being trained to segment the resealed image (20 a) into crop (11) and non-crop (12, 13) portions, and configured to provide a segmented output (20 s) indicating the crop portions (20 c) of the resealed image with pixels belonging to representations of crop; means to access a reference plot storage (20 cps) comprising one or more segmented reference images (20 cps 1, 20 cps 2, 20 cps 3) indicating crop portions (20 cpc) associated with one or more reference plots (10 - 2) in the agricultural field without herbicide application, the segmented reference images obtained by applying the image pre-processing module (120) and the CNN (CNN 1) to reference images (20 cp) representing real world situations of the corresponding one or more reference plots (10 - 2), with each reference plot being of approximately the same size as the test plot (10 - 1) and the one or more reference images (20 cp) being recorded under comparable conditions as the test image (20); a biomass reduction measurement module (140) configured to determine a biomass reduction measure (141) for the at least one crop plant by determining a ratio between the number of pixels in crop portions associated with the test plot and the number of pixels of crop portions associated with the one or more reference plots wherein, in the case of at least two reference plots, the ratio is determined by averaging over the reference plots."
  ],
  "cpc": [
    "G06T 7/11",
    "G06F 18/24317",
    "G06T 2207/10024",
    "G06T 2207/20021",
    "G06T 2207/20081",
    "G06T 2207/20084",
    "G06T 2207/30188",
    "G06T 3/4046",
    "G06T 7/0002",
    "G06T 7/0004",
    "G06V 10/454",
    "G06V 10/764",
    "G06V 10/774",
    "G06V 10/776",
    "G06V 10/82",
    "G06V 20/188",
    "G06V 20/70"
  ],
  "assignees": [
    "BASF SE"
  ],
  "filing_date": "2020-11-24",
  "publication_date": "2023-01-19",
  "priority_date": "2019-12-03",
  "application_number": "US-202017779819-A",
  "family_id": "68766681",
  "citations": [
    "US10713542B2",
    "US11073505B2",
    "US11093745B2",
    "US11263707B2",
    "US2019259108A1",
    "US9898688B2"
  ]
}