[WIP]add the confidence interval computation #387

YazhiGao · 2018-08-14T18:14:02Z

this is currently the implementation of ratio modeling for feature selection of random effect in photon.
I follow the algorithm described in the original publication but some twists are made according to discussion with yiming and alex.

Unit tests all pass.
Integration tests all pass.

The algorithm in reality(highly related with codebase instead of only mathematical expression) is as follows:

1.pass in the featureStatisticSummary
2.identify the binomial columns
3.compute the lowerbound for binomial columns based on the t value
4.select the feature based on only the following lowerbound criterion(non-binomial and intercept columns are kept automatically)

if (t < 1) {
  T_l = 1 / T_u
}

if (T_l > 1D) {
  //  select feature
}

As a WIP commit, there are things to polish in near future since we currently focus on the feasibility of this experimental method and try to minimize user-side changes :

unit tests not fully covering all scenarios of feature selection. Currently the binomial cases are not selected, we need to craft some data that covering all cases.
binomial feature column identification predicate needs to be stronger. Current solution is inherently flawed, we need more computation at feature summary stage to ensure this one.
hyperparameter interface design. for convenience purposes, the user side interface for pass in normal distribution quartile and lowerbound threshold hyperparameter redesign.
the relationship with pearson correlation feature selection. We need another parameter to decide on the algorithm of feature selection or mix them in later stage.
crafted test data need some change, currently some unneeded feature summary entries are not carefully addressed.
further experiment report and benchmark report after regression tests
the way we currently keep non-binary and intercept columns is not good for further feature ranking report planned. need redesign

@joshvfleming @ashelkovnykov

cmjiang · 2018-08-14T18:25:29Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

 import breeze.linalg.{SparseVector, Vector}
-
+import com.linkedin.photon.ml.Types.FeatureShardStatisticsMap


Add an empty line here.

cmjiang · 2018-08-14T18:25:48Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/RandomEffectDataSet.scala

 import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.rdd.RDD
 import org.apache.spark.storage.{StorageLevel => SparkStorageLevel}
 import org.apache.spark.{Partitioner, SparkContext}
-
-import com.linkedin.photon.ml.Types.{FeatureShardId, REId, REType, UniqueSampleId}
+import com.linkedin.photon.ml.Types._


Empty line.

cmjiang · 2018-08-14T18:26:52Z

photon-api/src/main/scala/com/linkedin/photon/ml/estimators/GameEstimator.scala

@@ -535,7 +541,11 @@ class GameEstimator(val sc: SparkContext, implicit val logger: Logger) extends P
            None
          }

-          val rawRandomEffectDataSet = RandomEffectDataSet(gameDataSet, reConfig, partitioner, existingModelKeysRddOpt)
+          val rawRandomEffectDataSet = RandomEffectDataSet(gameDataSet,


First argument in a new line.

joshvfleming · 2018-08-14T19:22:54Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+  def filterFeaturesByRatioCIBound(
+    intervalBound: Double,
+    percentage: Double,
+    globalFeatureShardStats: BasicStatisticalSummary): LocalDataSet = {


Args should be indented 4 spaces.

joshvfleming · 2018-08-14T19:23:03Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+  protected[ml] def computeRatioCILowerBound(
+    randomLabelAndFeatures: Array[(Double, Vector[Double])],
+    quartile: Double,
+    globalFeatureShardStats: BasicStatisticalSummary): Map[Int, Double] = {


joshvfleming · 2018-08-14T19:28:28Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+      randomEffectNumSamples += 1
+     features.activeIterator.foreach { case (key, value) =>
+       randomFeatureFirstOrderSums.update(key, randomFeatureFirstOrderSums.getOrElse(key, 0.0) + value)}
+    }


It would be faster / cleaner to do a vector sum reduce here, instead of manually iterating through the vector, e.g.:

val randomFeatureFirstOrderSums = randomLabelAndFeatures .map(_._2) .reduce(_ + _)

Also, generally speaking mutability is frowned upon in the Scala world.

joshvfleming · 2018-08-14T19:29:23Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+        upperBound match {
+          case Some(upperBound) => None
+          case _ => {
+            upperBound = Some(t * math.exp(math.sqrt(variance) * quartile))


This should also get its own function.

joshvfleming · 2018-08-14T19:29:45Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+        lowerBound match {
+          case Some(lowerBound) => None
+          case _ => {
+            lowerBound = Some(t * math.exp(-math.sqrt(variance) * quartile))


Same function as above.

joshvfleming · 2018-08-14T19:30:45Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+          case _ => py = Some(globalPopulationFirstOrderMeans(key))
+        }
+        val t = ( x / m ) / py.get
+        val variance = 1.0 / x - 1.0 / m + 1.0 / y - 1.0 / n


We should extract this out to a function that returns (mean, variance). There's too much going on here in-line.

joshvfleming · 2018-08-14T19:37:56Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+
+    randomFeatureFirstOrderSums.keySet.foreach { key =>
+      // Do computation on only binary and non-intercept features
+      if ((globallFeatureNonZero(key) * 1.0) / globalPopulationNumSamples == globalMean(key) && key != lastColumn) {


Let's put this in an isBinary detector function to make the code easier to scan. Also, this check for intercept is not quite right -- we might not have an intercept, in which case this will ignore an actual feature.

is this a temporary solution that i mentioned in to-do list. we will figure this thorough discussion.

joshvfleming · 2018-08-14T19:41:22Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    val labelAndFeatures = dataPoints.map { case (_, labeledPoint) => (labeledPoint.label, labeledPoint.features) }
+    val lowerBounds = LocalDataSet.computeRatioCILowerBound(labelAndFeatures, percentage, globalFeatureShardStats)
+    val filteredFeaturesIndexSet = lowerBounds.toArray
+      .filter(_._2 > intervalBound).map(_._1).toSet


Each of these chained calls should get its own line.

alice2008 · 2018-08-16T04:49:24Z

photon-api/src/test/scala/com/linkedin/photon/ml/data/LocalDataSetTest.scala

@@ -192,6 +248,61 @@ class LocalDataSetTest {
    }
  }

+  /**


comments style is

/** * **/

alice2008 · 2018-08-16T04:50:52Z

photon-api/src/test/scala/com/linkedin/photon/ml/data/LocalDataSetTest.scala

+    )
+
+    val computed = LocalDataSet.computeRatioCILowerBound(labelAndFeatures, 2.575, globalStats)
+    println(computed)


redundant println

alice2008 · 2018-08-16T04:55:36Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

@@ -126,6 +126,36 @@ protected[ml] case class LocalDataSet(dataPoints: Array[(Long, LabeledPoint)]) {
    }
  }

+  /**


comments are off. Same for line 245.

Should look like:

/** * text text text * */

This is resolved

alice2008 · 2018-08-16T05:00:08Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    *  Compute Ratio Confidence Interval lower bounds.
+    *
+    *  @param randomLabelAndFeatures An array of (label, feature) tuples
+    *  @param quartile The quartile score of standard normal distribution


what's quartile score?

This parameter should be called zScore, since that's what it is

mayiming · 2018-08-19T06:46:48Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    val randomEffectNumSamples: Long = randomLabelAndFeatures.length.toLong
+    val randomFeatureFirstOrderSums = randomLabelAndFeatures.map(_._2).toSeq.reduce(_ + _)
+
+    val m: Long = randomEffectNumSamples


m is not all the samples in random effect. it's the number of instance having binary feature f. x is out of this number m, how many are positive. Same for y and n for global.

@YazhiGao This one is my mistake: I misremembered the formulation when explaining it to you. Ping me if it's not clear what @mayiming means here.

ashelkovnykov

Please apply the Photon Style doc in the root of the project, it will reduce the number of style issues that need to be manually fixed.

Answering your 7 questions in order:

TODO - please fix the requested locations first and then we'll reconsider the unit tests
See comments
Ignore this for now - lock them the lower bound threshold to 1.0 and the z-score to 2.575. We can adjust them after PoC testing.
It's one or the other. During testing, we can compare how the Pearson correlation filtering performs. Based on past experience, I don't think it helps very much and I would personally prefer to remove it entirely. This would be in a separate commit.
See 1
After unit tests have been resolved we can introduce logging code for testing purposes
See comments

Thanks

ashelkovnykov · 2018-08-16T17:14:32Z

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

+    featureShardStats match {
+      case Some(featureShardStats) => featureShardStatsInMap = Some(featureShardStats.toMap[FeatureShardId,BasicStatisticalSummary])
+      case None => None
+    }


In Scala, var means variable and val means constant. As a general rule, var should not be used in Scala unless using a val would be inefficient or logically difficult.

Looking at how featureShardStats is used, this whole block should be scrapped and calculateAndSaveFeatureShardStats should be modified to return a Map instead of an Iterable.

ashelkovnykov · 2018-08-16T17:16:07Z

photon-lib/src/main/scala/com/linkedin/photon/ml/Types.scala

@@ -41,4 +43,6 @@ object Types {
  // A "feature shard" is an arbitrary set of "feature bags"
  // A random effect model corresponds to a single feature shard
  type FeatureShardId = String
+  type FeatureShardStatisticsMap = Map[FeatureShardId, BasicStatisticalSummary]
+  type FeatureShardStatisticsMapOpt = Option[FeatureShardStatisticsMap]


These types should be scrapped. The FeatureShardStatistics type in the GameTrainingDriver should replace Iterable with Map (see comments on GameTrainingDriver).

ashelkovnykov · 2018-08-16T17:17:06Z

photon-api/src/main/scala/com/linkedin/photon/ml/estimators/GameEstimator.scala

@@ -15,17 +15,15 @@
 package com.linkedin.photon.ml.estimators

 import scala.language.existentials
-


This whitespace should not be removed

ashelkovnykov · 2018-08-16T17:17:12Z

photon-api/src/main/scala/com/linkedin/photon/ml/estimators/GameEstimator.scala

 import org.apache.spark.SparkContext
 import org.apache.spark.ml.param.{Param, ParamMap, ParamValidators, Params}
 import org.apache.spark.ml.util.Identifiable
 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.DataFrame
 import org.slf4j.Logger
-


This whitespace should not be removed

ashelkovnykov · 2018-08-16T17:17:23Z

photon-api/src/main/scala/com/linkedin/photon/ml/estimators/GameEstimator.scala

@@ -43,6 +41,7 @@ import com.linkedin.photon.ml.supervised.classification.{LogisticRegressionModel
 import com.linkedin.photon.ml.supervised.regression.{LinearRegressionModel, PoissonRegressionModel}
 import com.linkedin.photon.ml.util._

+


Extra whitespace

ashelkovnykov · 2018-08-21T00:03:43Z

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

@@ -441,6 +447,7 @@ object GameTrainingDriver extends GameDriver {
        .setCoordinateDescentIterations(getRequiredParam(coordinateDescentIterations))
        .setComputeVariance(getOrDefault(computeVariance))
        .setIgnoreThresholdForNewModels(getOrDefault(ignoreThresholdForNewModels))
+        .setFeatureStats(featureShardStatsInMap)


We should only set this if the training task is logistic regression.

ashelkovnykov · 2018-08-21T00:07:38Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    val t = ( x / m ) / py
+    val variance = 1.0 / x - 1.0 / m + 1.0 / y - 1.0 / n
+    (t,variance)
+  }


py should not be an input

ashelkovnykov · 2018-08-21T00:10:25Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+      quartile: Double
+  ): Double = {
+    t * math.exp(-math.sqrt(variance) * quartile)
+  }


Unnecessary braces

ashelkovnykov · 2018-08-21T00:10:35Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    quartile: Double
+  ): Double = {
+    t * math.exp(math.sqrt(variance) * quartile)
+  }


Unnecessary braces

ashelkovnykov · 2018-08-21T00:12:37Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+    key: Int
+  ): Boolean = {
+    (globallFeatureNonZero(key) * 1.0) / globalPopulationNumSamples == globalMean(key)
+  }


This function shouldn't need to exist (see other comments), but if it did:

Passing an Array and an index to access within the Array is poor craftsmanship.
Why multiply by 1D?
Unnecessary braces
Need whitespace after function definition

this function is necessary as Josh pointed out to not write too much inline. As we further refine our predicate, we can do it faster.

YazhiGao · 2018-08-23T17:40:30Z

fixed issues but unit tests undone. early update for pending reviews.

YazhiGao · 2018-08-27T22:47:49Z

pushed with all tests passed and issues resolved.

photon-api/src/main/scala/com/linkedin/photon/ml/data/RandomEffectDataSet.scala

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

joshvfleming · 2018-08-28T14:04:46Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/RandomEffectDataSet.scala

+    featureShardStatsOpt match {
+      case Some(featureShardStats) => {
+        val featureShardId = randomEffectDataConfiguration.featureShardId
+        val (binaryIndices, nonBinaryIndices) = filterBinaryFeatures(featureShardStats(featureShardId))


"Filter" isn't a good name for this, because it's not really filtering a collection. When you see filter, you expect it to be (seq in, seq out).

@joshvfleming segregateBinaryFeatures?

@YazhiGao We can get around the intercept issue here, for now. Since we still need to run experiments, this code doesn't need to be 100% production ready yet, so what we can do is this:

val (binaryIndices, rawNonBinaryIndices) = filterBinaryFeatures(featureShardStats(featureShardId)) val nonBinaryIndices = rawNonBinaryIndices + globalFeatureShardStats.mean.length

Since we know that the intercept index is always the last index (for now), we can insert it into the set of non-binary indices to guarantee that it is not filtered.

I spoke with @joshvfleming offline and we agreed that we need a refactor to move the concept of the intercept into the GameEstimator, so this should be a reasonable hack for now, until that refactor takes place.

photon-api/src/main/scala/com/linkedin/photon/ml/data/RandomEffectDataSet.scala

ashelkovnykov

Have not reviewed test cases yet.

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

photon-lib/src/main/scala/com/linkedin/photon/ml/stat/BasicStatisticalSummary.scala

photon-lib/src/main/scala/com/linkedin/photon/ml/Types.scala

photon-lib/src/main/scala/com/linkedin/photon/ml/stat/BasicStatisticalSummary.scala

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

ashelkovnykov · 2018-08-28T19:44:20Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+      }
+    }
+
+    lowerBounds


As I mentioned in my previous review, we don't need all of these cases. We don't have to apply the paper exactly as it is written, since we have a special case where X ⊂ Y, always.

This entire block, from line 281 to line 316 can be replaced with the following:

binaryIndices .map { key => val x_col = x(key) val m_col = m(key) val y_col = y(key) val n_col = n(key) val lowerBound = if (y_col == 0 || x_col == 0 || x_col == m_col) { 0D } else { val (t, variance) = computeMeanAndVariance(x_col, m_col, y_col, n_col) if (t < 1D) { 1D / computeUpperBound(t, variance, zScore) } else { computeLowerBound(t, variance, zScore) } } (key, lowerBound) } .toMap

we shouldn't add the x_col==m_col as the condition for 0D output right?

after some thinking, I think the original code is right. We should only eliminate the x!=0 && y==0 case. this is the only impossible case in our data. The original code has excluded that.

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

ashelkovnykov · 2018-08-28T21:38:13Z

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

+            case None => featuresIndices
+          }
+          featuresAsBV.map{v => v(featuresIndicesWithoutIntercept.toIndexedSeq).toVector}
+        }


This intercept issue has become larger than we thought. As mentioned elsewhere, we should just hardcode the logic for now and return to it in a separte intercept refactor. I propose we use this block of code for now:

private def calculateStatistics( data: DataFrame, featureIndexMapLoaders: Map[FeatureShardId, IndexMapLoader]): Map[FeatureShardId, BasicStatisticalSummary] = featureIndexMapLoaders .map { case (featureShardId, indexMapLoader) => // Calling rdd explicitly here to avoid a typed encoder lookup in Spark 2.1 val featuresAsBV: RDD[BreezeVector[Double]] = data .select(featureShardId) .rdd .map(x => VectorUtils.mlToBreeze(x.getAs[SparkMLVector](0))) val featuresForSummary = indexMapLoader .indexMapForDriver() .get(Constants.INTERCEPT_KEY) .map(_ => featuresAsBV.map(dropIntercept)) .getOrElse(featuresAsBV) (featureShardId, BasicStatisticalSummary(featuresForSummary)) } private def dropIntercept(baseVector: BreezeVector[Double]): BreezeVector[Double] = baseVector match { case dense: DenseVector[Double] => new DenseVector[Double](dense.data, dense.offset, dense.stride, dense.length - 1) case sparse: SparseVector[Double] => new SparseVector[Double](sparse.index.filterNot(_ < sparse.length - 1), sparse.data, sparse.length - 1) }

what if there is no intercept?

how can the dropIntercept handle the case with no intercept?

It only gets called if there is an intercept.

YazhiGao · 2018-09-07T22:26:54Z

unit tests and integration tests passed.
rebased #390 to this commit.

YazhiGao · 2018-09-07T22:30:12Z

Should we take the case where x==m && y==n seriously like the original paper did to approximate the result using x = m -0.5, y = n - 0.5 or follow @ashelkovnykov 's opinion that this feature intuitively useless and we should output 0 lowerbound.
@mayiming @joshvfleming

ashelkovnykov · 2018-09-10T19:38:57Z

@YazhiGao @joshvfleming @mayiming

I'm set in my position that following the paper 100% is incorrect, due to our specific application:

X	Y	lower bound
0	0	Hardcode to 0 - this feature is never used
x	0	Impossible since X ⊂ Y
m	0	Impossible since X ⊂ Y
0	y	Regular case - need special value so no 'divide by zero' error during variance computation
x	y	Regular case
m	y	Regular case
0	n	Impossible since X ⊂ Y
x	n	Impossible since X ⊂ Y
m	n	Harcode to 0 - the lower bound approaches 1 from the bottom, thus feature will never be selected

Thus, the code can looks something like this:

if (y == 0 || y == n) {
  0
} else {
  val x = max(x_raw, X_MIN)
  ...
}

The X_MIN in the paper is 0.5 - this value seems arbitrary and I wonder whether or not we should decrease it.

joshvfleming · 2018-09-10T20:09:13Z

@ashelkovnykov is right. The formulation in the paper is between any two ratios, but in our case one set is a subset of the other. So there are cases that can't arise, and because of this, the code can be much simpler.

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

...n-lib/src/integTest/scala/com/linkedin/photon/ml/stat/BasicStatisticalSummaryIntegTest.scala

photon-lib/src/main/scala/com/linkedin/photon/ml/stat/BasicStatisticalSummary.scala

photon-api/src/test/scala/com/linkedin/photon/ml/data/LocalDataSetTest.scala

ashelkovnykov · 2018-09-12T17:51:11Z

photon-api/src/test/scala/com/linkedin/photon/ml/data/LocalDataSetTest.scala

+    )
+    val x_0_y_0_g_positive = Array(0.0)
+
+    // Second case x == 0 and y != 0


See my comments about splitting the responsibilities of the tests. The cases for testComputeRatioCILowerBound should be:

x = 0, y = 0

x = m, y = n

x = 0, 0 < y < n, epsilon = default

x = 0, 0 < y < n, epsilon = something else

x = m, 0 < y < n

0 < x < m, 0 < y < n

YazhiGao · 2018-09-17T17:26:38Z

fixed issues except for test cases redesign, all current tests passed.

ashelkovnykov

Ready for testing - @YazhiGao and I will work on the unit tests later, pending good experimental results from Baolei

photon-client/src/main/scala/com/linkedin/photon/ml/cli/game/training/GameTrainingDriver.scala

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

YazhiGao · 2018-09-19T20:53:18Z

ready for experiment now.

bl44 · 2018-08-31T18:09:36Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

@@ -212,6 +254,116 @@ object LocalDataSet {
    new SparseVector(filteredIndexBuilder.result(), filteredDataBuilder.result(), features.length)
  }

+  /**
+   * Compute Ratio Confidence Interval lower bounds.


Suggest to add more descriptions, and in particular reference the paper/book we refer to.

bl44 · 2018-10-04T18:04:05Z

photon-api/src/main/scala/com/linkedin/photon/ml/data/LocalDataSet.scala

+        val n_col = n(key)
+
+        val lowerBound = if (y_col == 0.0 || y_col == n_col) {
+          0D


for m_col = 0, we may also want to return 0D here. Since if it enter next branch it will produce Infinity.

cmjiang reviewed Aug 14, 2018

View reviewed changes

joshvfleming suggested changes Aug 14, 2018

View reviewed changes

YazhiGao force-pushed the ratio_random_effect_select branch 2 times, most recently from 070d006 to 93d78d3 Compare August 15, 2018 20:59

alice2008 requested changes Aug 16, 2018

View reviewed changes

YazhiGao changed the title ~~add the confidence interval computation WIP~~ [WIP]add the confidence interval computation Aug 16, 2018

mayiming reviewed Aug 19, 2018

View reviewed changes

ashelkovnykov suggested changes Aug 21, 2018

View reviewed changes

YazhiGao force-pushed the ratio_random_effect_select branch from 93d78d3 to 8e8eb55 Compare August 23, 2018 17:38

YazhiGao force-pushed the ratio_random_effect_select branch 2 times, most recently from 16e0374 to 6dace03 Compare August 27, 2018 22:46

joshvfleming reviewed Aug 28, 2018

View reviewed changes

ashelkovnykov suggested changes Aug 28, 2018

View reviewed changes

YazhiGao force-pushed the ratio_random_effect_select branch 2 times, most recently from 46fc425 to 3122532 Compare September 7, 2018 22:24

YazhiGao force-pushed the ratio_random_effect_select branch 2 times, most recently from 79d405e to 2e73bb6 Compare September 10, 2018 17:49

ashelkovnykov suggested changes Sep 12, 2018

View reviewed changes

YazhiGao force-pushed the ratio_random_effect_select branch from 2e73bb6 to e562b9b Compare September 17, 2018 17:26

ashelkovnykov reviewed Sep 19, 2018

View reviewed changes

add the confidence interval computation

ffc2a44

YazhiGao force-pushed the ratio_random_effect_select branch from e562b9b to ffc2a44 Compare September 19, 2018 20:52

bl44 reviewed Oct 4, 2018

View reviewed changes

		import breeze.linalg.{SparseVector, Vector}

		import com.linkedin.photon.ml.Types.FeatureShardStatisticsMap

@@ @@ -192,6 +248,61 @@ class LocalDataSetTest { @@
                   }
                 }
+                /**

		@@ -15,17 +15,15 @@
		package com.linkedin.photon.ml.estimators

		import scala.language.existentials

		@@ -43,6 +41,7 @@ import com.linkedin.photon.ml.supervised.classification.{LogisticRegressionModel
		import com.linkedin.photon.ml.supervised.regression.{LinearRegressionModel, PoissonRegressionModel}
		import com.linkedin.photon.ml.util._

[WIP]add the confidence interval computation #387

Are you sure you want to change the base?

[WIP]add the confidence interval computation #387

Conversation

YazhiGao commented Aug 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashelkovnykov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YazhiGao commented Aug 23, 2018

YazhiGao commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashelkovnykov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YazhiGao commented Sep 7, 2018

YazhiGao commented Sep 7, 2018

ashelkovnykov commented Sep 10, 2018

joshvfleming commented Sep 10, 2018

Choose a reason for hiding this comment

YazhiGao commented Sep 17, 2018

ashelkovnykov left a comment

Choose a reason for hiding this comment

YazhiGao commented Sep 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YazhiGao commented Aug 14, 2018 •

edited

Loading